Tag Archives: Python

Reinventing a wheel again: tSQLike

Posted on by
Reply

tSQLike is my Python 3 module to work with tabular data using SQL approach with such “primitives” as “select” or “join”. The idea was to make the library small and useful, while for the rest of tasks you can still use Pandas or NumPy.

from tsqlike import tsqlike

t1 = tsqlike.Table(data=[['h1', 'h2', 'h3', 'h4'],
                        ['a', 'b', 'c', 'd'],
                        ['b', 'c', 'd', 'd'],
                        ['f', 'g', 'h', 'i']],
                   name='first')
t2 = tsqlike.Table().import_list_dicts(data=[{'h1': 1, 'h2': 2, 'h3': 3},
                                            {'h1': 'd', 'h2': 'e', 'h3': 'f'}],
                                       name='second')
t3 = t1.join(t2, on='first.h4 == second.h1').select('*').order_by('second.h2', direction=tsqlike.ORDER_BY_DEC)
t3.write_csv(dialect='unix')

"first.h1", "first.h2", "first.h3", "first.h4", "second.h1", "second.h2", "second.h3"
"b", "c", "d", "d", "d", "e", "f"
"a", "b", "c", "d", "d", "e", "f"

Installation is simple as:

pip install tsqlike

Beating eval() slowness in Python

Posted on by
Reply

Turned out, eval() function, which evaluates and executes the specified expression, is dramatically slow in Python, as the compilation is done on the fly. Seeking for a way to enhance the performance, I have stumbled at the Stackoverflow discussion which highlights two key methods to speedup the process:

  • compile()
  • lambda()

Experiments with it show, that combination of both methods gives the best results:

Needless to say, my script, which used to take 20 minutes to complete, now ends in 50 seconds.

Doing magic with a self modifying Python script

Posted on by
Reply

The full circle of transformation: Original Script -> Changed Script -> GIF -> Original Script. See sources here.

NoneType issues in chains of inherited dictionaries in Python

Posted on by
Reply

There are good hints to avoid errors like:

AttributeError: 'NoneType' object has no attribute 'get'

in the brilliant Python tips 1: beware of Python dict.get() post.

An elegant way to limit concurrency with Python asyncio

Posted on by
Reply

Asynchronous code can sometimes be confusing because it tends to grow and becomes complex after some time. Fortunately, such a gem can save a lot of time and effort:

async def gather_with_concurrency(n, *tasks):
    semaphore = asyncio.Semaphore(n)

    async def sem_task(task):
        async with semaphore:
            await task
    await asyncio.gather(*(sem_task(task) for task in tasks))

From: https://stackoverflow.com/a/61478547/9127614

Voice Control for Vernie, the robot based on LEGO

Posted on by
2

Yes, it’s me who spends long winter holidays playing with children’s toys. We bought a wonderful LEGO BOOST creative toolbox for my son, but I insidiously picked it from the child and started enhancing it with Python to add Voice Control to order the robot simple movement commands. Sometimes it obeys my Russian English, but not always 😉

And “yes”, no Internet connection is required for the Vernie voice control to work 😉

Sources and installation instructions are on Gitlab. Feel free to message me on any issues.

My 2018 as a free software developer

Posted on by
Reply

The year is almost near the end and I’m feeling Christmas in the air already, now it’s time to recall what was done and count some results.

So, the first half of the year I used GitHub as a platform to store my code and manage updates.

My GitHub stats

As you see, I didn’t do many commits every day. That’s because I’m too lazy I still prefer developing old-school ways (yes, blame me for that 🙂 ) and keep all changes on my PC, pushing only valuable changes to public repositories. Also, I mostly code in my free time: in the evenings and on vacations, therefore you can see empty gaps between commits.

The other reason is that my favourite the BeaST project has a main page only on my blog and the BeaST Quorum (BQ) source code on SourceForge platform.

In June I migrated most part of my code to GitLab, though I didn’t also delete the GitHub account for references and history.

My GitLab stats

So, what projects can I mark “successful” this year? Actually, I’m looking sceptically at all of them, but at least I feel less shame mentioning these projects, code and activities:

  • The BeaST storage system at Open Analytics – a consulting company specialised in supporting the data analysis process of its customers end to end. Together, we have launched tests of the BeaST on a real bare-metal hardware. And it works! After functional tests the BeaST is going to be a LUNs provider for the Kubernetes cluster of the Open Analytics testing environment. I promise to write a post (or two) about this significant milestone in the life of the BeaST project.
  • svcstats – a CLI stat-like utility for reporting IBM SVC/Storwize storage system performance statistics using SMI-S interface. Though I started coding it in 2017, a lot of work has also been done this year.
  • sp_ping – my own simple ICMP ping implementations in Python. It supports both IPv4 and IPv6 protocols as well as unicast/broadcast/multicast requests. It’s code I later reused in the Net Radar – Simple IP networks browser software, which is currently in the early development stage.
  • ds8k_perfs.py – Another stat-like utility. It displays Open-systems performance metrics for IBM DS8000 storage system in CLI. Utilises SMI-S protocols as well.
  • supermatic – a simple HTTP server I wrote just for fun as Bourne Shell script. Yes, It’s a toy, but who cares? 🙂

IPv6 multicast ping in Python

Posted on by
1

In the meantime, I continue to reinvent the wheel: just added IPv6 multicast ping to my small ICMP script collection in Python.

Usage:
	sp_mping6.py -i interface [-a address] [-m ttl] [-t timeout] [-v]

Options:
	-i				interface Interface to work on
	[-a address]	IPv6 multicast address to ping or ff02::1
	[-m ttl]		TTL of outgoing packets. Default is 64
	[-s number]		Packet sequence number: 0-65535. Default is 0
	[-t timeout]	Timeout on socket. Default is 5
	[-v]			Be verbose (show more info about packets)

Note! You must be root to run this program.

And finally, sp_ping project page is here. Sources are there.

My ICMP ping supports IPv6 now

Posted on by
Reply

Wohoo! This Sunday was awesome: I have just added IPv6 support to my own ICMP ping implementation in Python 3. Amazing, but it works:
I’m not sure about Linux, but it successfully pings Google from my small FreeBSD box 🙂

A simple way to visualize IBM SVC/Storwize performance

Posted on by
2

Recently I wrote a set of tools to collect and display in CLI performance statistic of IBM Spectrum Virtualize: SVC and Storwize systems. Using various protocols it’s possible to get both detailed performance for each vdisk, mdisk or drive as well as generalized statistics for the entire system.

These textual outputs can be pushed into any monitoring systems like Zabbix. But one of the easiest way is to use famous RRD to accumulate data and build such graphs. Below is a screenshot of an HTML dashboard with Controller-level performance statistics for a single IBM SVC system:IBM SVC/Storwize performance dashboard

Such simple dashboards and diagrams can easily be created for every recourse of the storage system. An example of the script for creating this controller-level performance dashboard is included into svcstats.py distribution under 2rrd directory.