State of libwaveform after GSoC

Hi everyone, this is long waited status update of libwaveform - library, who aims to provide all necessary tools to collect, store and draw waveform data. After dealing with studies I have started to hack again. First, more about where I was at the end of the summer - I had general reading part aka WaveformReader ready, using Gstreamer level element, and also generic waveform drawing widget WaveformWidget using gathered data. However drawing just was without any zoom support. So first thing I wanted to improve and work on post summer was to provide method how to deal with zooms.

First of all I started to think about various ways how zoom level can be measured. I went wrong direct several times, but in the end I settled for same old 'pixels per second'. So for WaveformWidget object I have variables which store current, default and max/min zoom levels by this method, and methods which increases and decreases zoom level - currently by simply multiplying and dividing by two - which can be called using callbacks from clicking on buttons, for example (see demo/demo.py to test this out). Now when I have changeable zoom level, I must decide how to draw waveform in given situation. This is what I have in mind for now.

For example, if I have waveform data reading interval 0.1 second long (or 100 000 000 ns), and I have 10 pps (pixel per second) zoom level, that means that wave will have width 1 pixel. That is not enough in a case if I have a waveform (usual one) where minimum width of wave side is 2 pixels - so I will require reading interval 0.2 seconds long from WaveformData (because of it's next reading interval upwards we calculate adding two 0.1 readings together), and we will use step of 2 pixels (because 1 sec /0.2 sec per reading is 5 readings and 10 pixels/5 readings is 2 pixels for reading).

This works nicely if I have waveform max and min width in sync with zoom levels/steps. However, I'm still struggling with scenario when you have 2.5, 3.13, e.t. pixel width if you have some odd zoom levels (for example 71 pixels/sec). There are two scenarios - either I enforce particular zoom levels, calculated "next best thing" by given numbers from user (similar like size request in Gtk+), or just warn users about not having pixel precise waveform alignment. For now I will work on second one.

Next big target is to fix Gstreamer level element to have sub-buffer intervals, otherwise for wav file instead of 100 000 000 ns interval I get 120 000 000 ns - and so on. Currently code has condition test for given interval size, but it doesn't work and actual interval gets rounded by buffers n times.

For feedback first my question is - what do you think about zoom model I offer here and how I plan to draw waveform according to it? What could be potential pitfalls (I have thought of several corner cases, but fresh view wouldn't hurt). Also I would like to know common values you use for default, min and max zoom levels.

How to get libwaveform - current development branch is 'postsummer' on https://github.com/Pecisk/libwaveform. It's buildable on Fedora 18 without any big dependencies (you need usual suspects from Gtk+/Glib land, also gobject-introspection-devel if not mistaken), just use --prefix=/usr with ./autogen.sh so it would be installed correctly in your system. For other systems probably best is to use/install it with jhbuild. To check out recent functionality see demo/demo.py.

Libwaveform final Google Summer Of Code report

Finally it's a last day of this year's Google Summer Of Code - it's when I need to calm down, look back what I have done and what's still in my todo list. As I plan to carry on with developing this library, end of summer project is merely a milestone, albeit very important one.

Before this summer I had been working with Gtk+ and GObject/GLib on various projects (but mainly on Jokosher, which I try to continue to port to Gst 1.0), however never so seriously as in this project (and never using C). It took some time to get full swing of how things are done in C/GLib/GObject combo. As usual, C requires no taking things for granted, as usual successful compiling doesn't mean it works really as it should. Seriously improving my knowledge of C, GObject, Gtk+ and how to debug all this is my major gain from this project.

My project aim was/is to create usable set of elements which together would allow provide a widget which would draw a waveform. Those elements are WaveformReader, WaveformData and WaveformDrawing (as I have named them in my code for now). Reader handles reading of the sound levels from files (also decoding them accordingly, all using Gstreamer 1.0), Data provides data model and method to add information to it (so far), and Drawing is a generic widget (derived from GtkDrawingArea) which goes trough data in data model and draws it using cairo library.

I have done most of stuff I initially planned for Reader, however new ideas and issues popped up. I have one method to read levels, which takes source URI, interval, start and finish time as parameters, which works as expected. However while testing this I got unexpected behaviour from 'level' element - as I found out, it can't parse data for smaller interval than buffer, so if you even set interval different than default, you can't expect to get results precisely for that buffer size. This was something too hard to fix for me during project time, but I plan to do so with a help from my mentor Sebastian. I also need to expand 'level' element with having sort of 'lowest peak' value for buffer, as it helps to draw nicer waveforms. This also is on my todo list for post-summer coding.
For error handling in library I choose GError. For Reader I added several errors and have tested it trough Python exception handling - you can see example in demo/demo.py script.

I had lot of ideas and theories how data model should look like, however I settled for quite simple now (see PDF here, I will add more improved version later). Most difficult part is related how to treat more detailed data which are needed for zooming in. I'm planning to return to improving data model when I will get fully working stack within theoretical application (like Pitivi or Jokosher). I have started to create TODO file, which naturally and slowly turns into design document. Currently Data has adding method which detects invervals and time and put readings in right places. For storing readings for each channel I use boxed structure (which is quite buggy I admit and will get cleanup treatement postsummer) named WaveformLevelReading. For GI annotations GList of readings is never transfered fully, as it's quite a chunk of data and copying it wouldn't be very smart resource wise. In current state of things Reader create GList of readings, and when adding it to data model, it practically becomes a part of data model, without copying.

For Drawing - widget part - I first stuck at creating widget itself as an object in Python. For some strange reason manually written annotation with (transfer full) for constructor method failed and object weren't transfered, ending it's run into segfault wall in next second I was trying to access it. Thankfully, just removing annotation worked. It was strange as it was copy pasted from other object were it worked. After that I tried to follow instructions from various tutorials - ways to create custom GtkWidgets have changed considerably over years, so probably it would be wise to create some general intro tutorial for doing it in C/Vala/Python - and finally succeeded in having created object which didn't segfaulted and which I could add to generic GtkWindow. After that I started to dig into Cairo stuff, which resulted in very simple waveform which I have now. I tried to replicate Jokosher waveform style with bottom part filled up, but didn't succeeded so far, I will have to look at that later. It's clear that widget part will need most work with style support, customization, and mostly allowing create subclasses  so people can draw overlays. Post-summer my first job will be making zooming work, also implementing async methods of updating waveform data and drawing.

As usual, biggest issues were theoretical ones - understanding how some of more complex stuff work required more time than I planned to. However acknowledging that it's better to dig trough them now than later I did several rewrites and studied lot of Gstreamer code (thanks to Sebastian about giving references and reviews with comments), as I will need to provide patches to Gst or understand how to write async methods. I also have to admit that I was overoptimistic about project's timetable - as usual, studies made things difficult at the beginning of the summer.

In overall, it was very refreshing experience (although I didn't finish lot of things I wanted to - especially implementing zoom logic) and I'm happy that I got some foundations (and beefed up knowledge) in place so I can continue to hack on this library. In more general sense it was definitely worth it because of increased knowledge which will help me to grow as a coder and will enable me to help with code and patches other projects like Gstreamer and Gtk+ in the future.

Project can be found in github.com/Pecisk/libwaveform, see 'postsummer' branch for futher changes for now (as long as I submit code to Google). I will post more updates after week break which I need for my studies. Stay tuned.

GSoC 2011 Jokosher porting to Gtk+3/PyGI - status quo

Yeah, this year again I'm working on Jokosher during Google Summer Of Code. Last summer I tried to implement Telepathy calls, thus creating nice podcast tool. While I partly succeeded and anyone willing to test it can do it using this Jokosher branch of mine on Launchpad, it . I'm planning to return to it after my porting job this summer because I want to see that feature in master branch and official release.
This summer I have different, but yet still very interesting task - porting PyGTK stuff over PyGI. GI comes here as GObject Introspection. To get full details about GI read here, but in nutshell it is method to create mostly automated bindings, using special API scanners and creating Python counterparts of C methods and variable types. Therefore PyGI API follows very closely C API. This gives two very obvious advantages - there is no need of complex support of bindings, and they are always mostly up to date.
For Jokosher it means that I have to replace most of PyGTK syntax which differs from Gtk+3 C API syntax. Thankfully, pygobject hackers and other coders with their porting tasks have created nice bash script called pygi-convert.sh which will replace lot of the obvious stuff automatically (like 'import gtk' to 'from gi.repository import Gtk'). While it helps porting considerably, you must take into account that it is quite "dumb" and can create some nasty side effects, especially on such large app as Jokosher. So, I started with pygi-convert.sh and then ran Jokosher in Gtk+3 enviroment to get errors, fix them and repeat these steps as much as needed. Mostly it was when script had done something wrong, like inserted replacement strings nested twice, but mostly it went surprisingly smooth.
After that I had usual "trial and error" sessions. List of bugs I encountered doing this I have put online here - and actual code can be found here, in my project's Bazaar branch on Launchpad. Taking into account rather nasty exams I had this year I'm quite happy with what I have achieved so far. My little but effective knowledge about Gtk+ internals is helping me considerably - and again big thanks goes out to hackers at #gnome-hackers and #python channel at GimpNet. Biggest problem so far I faced with is waveform drawing. Jokosher doesn't issue an error or traceback, but it doesn't draw a thing neither. So it doesn't let me try out event manipulation either, so I could have to check out considerable size of Gtk code after fixing waveform.
So for second part of this project my major aim will be to fix this waveform drawing. I also have bunch of small errors (like Unicode problems) I have yet to figure out how to fix - or they just side nuances of using jhbuild enviroment. After this I want to try out moving to PyGI for Gstreamer. This probably will be very difficult but also fun, because no one have actually tried that at home yet.

In the end - final report of my GSoC

Hi everyone!

It's have been great 12 weeks. While I failed to complete half of my plan (Telepathy Tubes support), I got working VoIP call recording - which was a reason why I get involved in this Google Summer of Code in first place. But before that I have little story how I got there. For just to see how it works, see this video I made during conversation with friend of mine (sorry for faulty cheap headset, which forced to cut middle section of conversation with Pitivi, as headset just died on me). You can see me running Jokosher, choosing contact, doing recording and pressing stop and getting waveform redrawn, and in the end doing playback of recorded conversation.

First of all, I got old Michael Sheldon's code in which he tried to get all accounts, connect them, and then you could choose contact and call him (or her). Code has been rotten for some time in Launchpad branch, and Telepathy has changed lot of things here and there. First of all, I run old branch and saw that nothing worked. Piece after piece I gathered information how to connect to network, how to get contact handles, etc. People at Telepathy IRC channel (#telepathy at Freenode) were very helpful. Also I have to give thumbs up for Telepathy spec documentation - if you start to understand concepts, it is mostly all you will need. Also Python bindings have excelent examples.

In fact, things change so fast in Telepathy world, that I had to build newer Ubuntu package for Python bindings, as Ubuntu Lucid one lacks Account Manager support. With a tip from Michael I made newer one using 'checkinstall' program, which allows fast and dirty Debian package building. If you plan to test my code in Lucid, download it from here and install it over your current version. If you want to build it yourself, get a git repository checkout, build it, execute 'checkinstall' in bindings directory, change settings (name of package, what does it provide, etc. - make sure they fit current naming/version scheme of python-telepathy package) and run it. It will build a package and install it. For other distros just be sure you have newest Telepathy and it's Python bindings installed. Additionally, for all distributions you will need Farsight Python bindings and Farsight/Telepathy glue library bindings too (named python-farsight and python-tpfarsight in Ubuntu).

After completing basic connection and retrieving information about accounts, I started to work on doing actual call. At first, I simply hooked autoaudiosrc and autoaudiosink as way to hear and record for call. It was first time I stuck, because implementing a call properly within Jokosher required to understand what actually happens there. In the end, it was very important to add watch for Gstreamer pipeline bus so Telepathy/Farsight would know what happens and could act on that. After that I could do calls, but there was no recording done.

For actually recording it, I had to join all Farsight pipeline stuff with Jokosher inner recording system. Jokosher records each instrument using separated bin, which are put together in one big pipeline (allowing to set states in masse). Not wanting to complicate things, I just extended current Farsight pipeline with two tees and split output and input so I could record and send my voice to remote participant in same time - and I could record and hear voice of from other side too. Recording is done by similar bins as for normal instruments, just tweaked for VoIP streams. This is where I stuck quite seriously. In the end, actual solution to my problem was quite logical - Farsight sink pad for sending voice to remote end and source pad for getting it from there - gets created only after whole pipeline is set to playing state. That means that those pads expects any other element to be set to playing before it gets linked to it. Thanks Tester from #farsight@Freenode for explaining it.

Before SoC deadline I tried to fix all small things which have popped up during implementing call support. I failed to link VoIP pipeline into main one (so it would have proper refresh and waveform redrawn regularly), but it requires too much changing of current state, so I hope to deal with that together with Michael post-SoC.

So, for now I got a call recording working, but it lacks waveform redrawing during conversation, and all is done separately from main pipeline, so it is not perfect yet. However, it gets job done (today I recorded conversation with my friend for 30 mins), so anyone can get cracking recording their podcasts and meetings. First, make sure you have all dependencies I mentioned above. Second, I would suggest to use Pulseaudio to get software mixing of audio streams so nothing would block playback. Third, pull a code from my branch (bzr branch lp:~pecisk-gmail/jokosher/telepathy-ng). Then go to telepathy-ng directory and launch Jokosher with 'Jokosher/Jokosher'. Add VoIP instrument (VoIP Local will be added automatically), Then change playback system to PulseAudio (Edit => Preferences). After that, choose your conversation partner using 'File => Instrument inputs...'. After that press Record button and wait. As I haven't find a way to bind pipelines yet, there is no "Waiting..." message. When your partner will accept call, event for each instrument will appear. Then conversation (and it's recording) will be started. After finishing call just press stop to put pipeline into null state and redraw waveform for each file.

Please, report all bugs and suggestions to me, pecisk at gmail dot com, as code will have long way to be officially included in Jokosher, so let's not bother Launchpad with bug reports yet. I hope to get call support in user friendly state till the end of August so we can try to include it in newest Jokosher release.

Very long Google SoC report #3

Hi everyone!

I was skipping my blog as I was trying to get Telepathy calls working within Jokosher. I wanted to post something when it was working actually. Well, I haven't got to that point yet - as I still trying to connect Telepathy call stuff to Jokosher - but I feel obliged to tell you more how I got there, because it was quite interesting road so far.

First of all I tried to run Michael Sheldon's code and see what works and what doesn't. In overall, I tried to use newest Telepathy API spec, so first to get somewhere I needed to build python-telepathy package from newest GIT branch. I did it with checkinstall (found in Ubuntu universe repo, thanks to Michael about tip), which is quite nice tool for fast-and-dirty package building. In source directory just ./autogen.sh, build it with make, then launch sudo checkinstall. Just make sure to change version to something appropiate, like 0.16 and change Package name and Provides parameters to 'python-telepathy'. Then checkinstall will create deb package and will install it into your system, so get ready to get your system trashed a little bit (for python-telepathy it isn't such a trouble).

After that I submerged in world of VoIP. Thanks to guys at #telepathy at irc.freenode.net, I understood how telepathy python bindings works and got around implementing current code. Bigger problems raised when I tried to understand how Farsight kicks in during the call, but I got it too - I was lacking binding that would made Telepathy/Farsight listen to gstreamer bus messages and then act accordingly (in fact, I very good example how to make a call from python is in Telepathy Python bindings package, examples/call.py).

For last two weeks I have tried to bind TP/F pipeline with Jokosher recording system. As most of sinks and pads in TP/F conference/session is created on the fly, it is quite challenging. It works like this: I have created gobject signal for TelepathyContact to bind eveything with Jokosher pipelines in Project.RecordVoIP. When call is successfull, this method binds every piece to record it in instruments. I have two challenges here - I have to create another instrument besides voip to record local speaker (let's call it "voip-local") and have to find a way to sync Jokosher pipelines with call ones. Also I need to decide to I allow recording happen at once or wait for remote contact to respond.

This is midterm review week and while I have had some setbacks - family stuff like sister marrying (yay) and very hot last two weeks, which makes me feel exhausted - I think I will finish my primary goal this week and will start to work on pipes support. Current code in telepathy-ng branch at Jokosher LP isn't doing anything useful, due of me improving code every day, but you can check out and test it if you like.

Google SoC report #2 - Discovering MC5 and friends

Hi everyone, this is my second report about my Google SoC project, which will bring initial Telepathy support for Jokosher. In previous week I was very busy with my qualification paper, which I finished and submitted this monday. After that I jumped in.

First of all, I joined IRC channel #jokosher at irc.freenode.net (or irc.ubuntu.com, if you like) to discuss with Michael Sheldon and Laszlo Pandy how to practically achieve my goals. As Telepathy stack have been developed rapidly last several years and things have changed since Michael's work on his telepathy branch, I had to rewrite method which gets accounts from Telepathy first. Previously it used gconf to store account information, but now you have to do it all using D-BUS magic. Of course, there are lot of things made easier with using python-telepathy bindings. Telepathy guys are slowly moving focus to gobject introspection and pygi way of doing things, but more or less I and my mentor agreed that I have to move forward with former. I also changed application's visual behavior so in a case if there is no accounts of supported type defined and user tries to add 'VoIP' instrument, it gives nice information message in error message area. I also doing patch outside my scope (which I plan to finish tomorrow) of the project for supporting GTK InfoBar, as currently Jokosher use custom class to form this area, to fix various theme bugs - like current one with dark fonts on dark background. I also plan to add button to open Empathy Accounts dialog, so users can add accounts straight from Jokosher.

For next week I plan to connect all dots to actually record conversation. This will be interesting challenge, as I never fully investigated gnonlin. Another problem I want to fix is to get Jokosher connecting to accounts when Empathy already is using them. Currently it requires Empathy to be closed as it tries to request new connection.

Of course, all code can be found in lp:~pecisk-gmail/jokosher/telepathy-ng. You can access code from Ubuntu using command 'bzr branch lp:~pecisk-gmail/jokosher/telepathy-ng'.

Google Summer of Code 2010 and long time no post

Hi here!

I have several blogs on blogspot.com, but for very long time I was quite busy and had time for doing several posts only in Latvian ones (Mostly Ubuntu, free software and OpenStreetMap related). Now I finally have some time and what's important, reason to do more posting here, so I hope it will become more of regular habit. Reason of course it's quite thrilling - some time ago I was confirmed as participant of this year's Google's Summer of Code, which is very great, because I love this initiative and have always wanted to be part of it. So here I will try to detail my plans for my summer of code.

In nutshell, I will try to make Jokosher a podcast killer application. When Jono Bacon and friends initialized Jokosher, main aim was to make podcasts easy and without lot of technical wizardry. For now, Jokosher have nice basic recording capabilities, with lot of good LADSPA effects and of course with excellent user interface. In previous years, we have fought to have multichannel cards supported trough using ALSA Gstreamer elements and also having a basic stability so it wouldn't crash when something really bad happens.

For my work I will use Michael Sheldon's work on Telepathy VoIp support in Jokosher, so I have to create user interface, connect all dots and of course, test it in real life, doing real podcasts with friends :) Then I have two bigger challenges, first one I have to extend current code to use Telepathy tubes so I can do lot of cool things remotely within Jokosher-Jokosher session - and then I will use this tubes support to create high def sound sync - It will work like this. For communication I will use simple VoIP call. After call will be finished, Jokosher will sync actually recorded sound with Jokosher at the other end, so in the end, you can have very high quality mix for your podcast, while doing call using low bandwidth codec.

And yes, finally this is also my "hi" to Planet Gnome!