I bought kinect last weekend (right after releasing the official api) and I’m really amazed by its possibilities. New ideas keeps popping up each day, so stay tuned for more vids :)

Have you ever experienced this strange feeling when watching photos that you took last night during all-night-long acid trip? Suddenly you realize that the Eskimo you were talking to for couple hours is just a toilet seat. And the amazing cocktail you were sipping consists mostly of herbal shampoo with vodka and glass cleaner. I know how it feels like. But fear not – it’s just the logical part of your brain trying to trick you. This app will help you release psychedelia hidden in the photos and show everyone how it REALLY was. Stay away from “Trippy Inverter“!

This is a quick Windows Phone 7 application for image processing. Whole project took 1 day, half of which was setting up the dev environment. Main goal was to figure out the WP7 release process & collect stats – in case I’d like to make a more serious app.

You can also check that out from various download sites.

Now you are screwed with shitty design
Layers of crap with ain’t no plan
Frickin’ solution that just won’t scale
Countless regressions and no one care

Problem:

Splitting up handwritten calculations into single lines.

Algorithm description:

First map all the strokes on the y-axis. Then travel along the axis and record the the ink density (number of strokes underneath). Remember local maximums. If the density falls below a certain percentage (tolerance) of a last local maximum – insert a break line.

C# code:

            // Positions of line dividers
            List<double> LineDivisions = new List<double>();

            // Top & Bottom edges of each Stroke
            SortedDictionary edgePoints = new SortedDictionary();

            // How much of ink density do we tolerate between the lines?
            // This is usefull when the lines are overlapping
            // 0 means there must be a clear interval between the lines
            // 12% means that if the density goes below 12% of local peak then we're dealing with a line break
            // Too high value might result in false line breaks.
            double tolerance = 0.12;

            double lastDownhillPosition = 0;
            double densityLocalPeak = 0;
            double density = 0;

            foreach (Stroke stroke in InkCanvas.Strokes)
            {
                Rect rect = stroke.GetBounds();

                if (edgePoints.ContainsKey(rect.Top))
                    edgePoints[rect.Top]++;
                else
                    edgePoints.Add(rect.Top, 1);

                if (edgePoints.ContainsKey(rect.Bottom))
                    edgePoints[rect.Bottom]--;
                else
                    edgePoints.Add(rect.Bottom, -1);
            }

            // Travel the density map & mark the all the line breaks
            foreach (KeyValuePair edgePoint in edgePoints)
            {
                // This part is in front because we want to peek whether the next point will bring us high or low
                // If it goes lower, then we don't want to insert line break yet - let's wait...
                // If higher (edgePoint.Value > 0), that means we just passed a local minimum and that's a potential line break.
                if ((density < densityLocalPeak * tolerance)
                     && (edgePoint.Value > 0) // we just passed local minimum
                     && (lastDownhillPosition != 0)) // don't put the line break above the top line
                {
                    LineDivisions.Add((edgePoint.Key + lastDownhillPosition) / 2);
                    densityLocalPeak = 0;
                }

                density += edgePoint.Value;

                if (edgePoint.Value < 0)
                    lastDownhillPosition = edgePoint.Key;

                if (density > densityLocalPeak)
                    densityLocalPeak = density;
            }

            // mark the end of the document (usefull for further processing)
            if (InkCanvas.Strokes.Count > 0)
                LineDivisions.Add(InkCanvas.Strokes.GetBounds().Bottom + 4);

Sample results

My friend’s calculations (posted without asking for any permission :)
Used Tolerance level: 12%.
Note the ink density visualized on the left edge. For better overview a different color was applied for each line.

Sample A - quick line segmentation

Sample B - quick line segmentation

Comparison with Microsoft.Ink.Divider results

Since the API for equation recognition engine is not available in Win7 I used the regular handwriting processing library.
Results for the same samples (note the differences between various runs!):

Sample A - Microsoft.Ink.Divider run 1	Sample B - Microsoft.Ink.Divider run 1
Sample A - Microsoft.Ink.Divider run 2	Sample B - Microsoft.Ink.Divider run 2
Sample A - Microsoft.Ink.Divider run 3	Sample B - Microsoft.Ink.Divider run 3

The variations in outputs are probably result of indeterministic Neural Network algorithm. The method also does character analysis and most likely uses this context information too. All that funky magic costs quite some execution time – from 100 to 800 ms.

Algorithm described at start takes 1 – 2 ms for the same samples and frankly speaking I like it’s results more. The fact that it has adjustable tolerance level and provides consistent deterministic results makes me feel better too. However it still got some limitations:

• produces false break lines for small samples (i.e. for 1/2 fraction alone) – this can be mitigated by requiring a certain ink density level inside each line or taking height/width ratio into consideration
• won’t work for non-horizontal formatted writing – the document need to be normalized (rotated) correctly
• can’t handle scribble-styled calculations (i.e. many columns with each having different line structures) – would need to introduce vertical segmentation too

All of the above limitations can be easily fixed. The goal of this post was to introduce the idea in a form as simple as possible.

Recently I’ve started working on a new project with my friend, so development of Falasol will be slower for a while. But fear not – I’m not going to give up on that. Falasol is my long term plan and what you can see so far is just a little part of it. I expect to get back to it next year. Meanwhile I’ll be occasionally posting stories from EqOCR development. You probably guessed that it will have something to do with equations & math – that is correct.

It was back at school when I discovered the charm of equations

Since me & my friend are big fans of Fringe science & cutting-edge technologies we were excited to find out about Win7 Math Handwriting Recognition capabilities. After going through API overview we couldn’t wait to plug into win7′s recognition engine and feed it with a basic quantum death ray formula. Until we figured out, that uhm… there is no such engine…

plan: analyze a sheet of handwritten equations



Turns out that you can analyze text written in whole bunch of languages: English, Polish, Chinese, Klingon… But in order to take advantage of Math Recognition, you need to use Math Input control. OK then, we thought, let’s embed the control, pass the handwriting and retrieve the analyzed output. Guess what… this control can’t be embedded.

Math Input control pop-up window

Math Input control is so cool that no application is worthy hosting it. Math Input control can only be presented as a pop-up window. That would be fine for one or two lines, but our annihilation formula is one page long. And destroying the whole universe takes at least a dozen of calculation pages. Popping window for analyzing every single line? – no can do! Next idea: Let’s show the pop-up in a hidden state and pass the Ink written in our main window. Too bad this beauty won’t perform without audience… You need to call Show() prior to any other activities.

In the act of despair we considered displaying Math Input just for the while needed to programatically load the Ink & analyze it. Yeah, I know such flashing windows could cause epilepsy, but a true scientist should be used to all kinds of unexpected bursts and explosions. Yet another surprise… there is no GetResult() method present in the control’s API… That function is only provided via MCTFB (Manually Click The Freakin Button) event handler.

On top of that there is this shiny piece of COM object comes without any .net wrapper. And if you happen to use WPF you need to know that there are two versions of Ink object: full-blown-all-having COM Ink and it’s handicapped twin brother WPF Ink (with very limited set of methods, which – if you want to do any analysis – you need to convert to COM version anyway).

To resume: There only way to take advantage of Math Recognition in win7 is through a Math Input control. Which must be popped up right in the user’s face as a separate window. And requires manual confirmation whenever any analysis is done. I’m sure it does tremendous job with protecting the world from evil scientists (any scientists in fact), but you’ve got to do better than that to stop our crew :)


PS.

I do appreciate all the effort put in Math Input and it does a really good job in manual usage scenarios. This post was intended to show how frustrating it is not being able to use the engine programatically due to poor API support. I couldn’t find those limitations mentioned anywhere in the documentation, which gave the wrong impression about the tool’s capabilities (especially when compared to well designed regular handwriting recognition support).

Let’s go crazy without breakpoints
Strings like yours is all that geek wants

Check my bigint, in a while…
We’ll have private double func

In library
At run time
Let’s join threads and override
With your 0 and my 1
We’ll ignite a great bit bang!

Morning time-bomb

This setup was on my mind since I was a kid. But it was recently when I found the missing device to make it happen. Couldn’t believe when I saw it on the shelf at my usual shopping market: THE POWER STRIP TIMER.

So here it is: my geek’s dream wake-up scenario!

1. Evening

   • Set-up the turntable with an LP to be played in the morning
     Can be classical Vivaldi’s Four seasons, jazzy Amy Winehouse, Sabaton‘s battle-metal, chilled out Air, Fink Flooyd‘s psychedelia, dark vibes of Black Sabbath or what have you…
   .­
   • Start a quiet playlist from the laptop for good night via surrounding 5.1 speaker system
     Before falling asleep I usually have some wine & read a book in bed for a while. It’s nice to have something playing in the background. Happened few times that a song was able to ‘incept’ a cool dream :)

2. When sleeping

   The scheduled task on the laptop stops the lullaby music around 1.30 – 2 o’clock. REM sleep phase is better with as little outside stimulus as possible.

3. Morning

   7.30 – power cord timer kicks in

   • Sun-imitating lamp gently warms up
   • Vinyl gets a spin – for an extra kick the turntable is connected directly to amplifier =:]
   • After soaking up the music for a couple of minutes ain’t nothing gonna break my stride!

The LP usually finish about the time I leave. Automatic timer turns-off the lamp & turntable few minutes after.

How cool is that! :D

Music of the prime numbers

Each harmonic corresponds to one prime number. A note is played on its beat-multiplication

Zipped project

This idea was on my mind long before starting the whole Falasol project. What kind of cacophony would it be if each note is assigned a prime number and played only on the number’s multiplication beats. Total chaos? Or would the mind be able to recognize some hidden patterns?

It’s both. The brain can grasp 2 & 3-beat notes and arrange them into 6-beat tact ( 2 * 3 = 6 ). Next 5-beat harmonic creates a 30 beat verse (consisting of 5 described tacts). That’s the rhythmical clocking in the background. Next prime harmonics (7, 11, 13, 17, 19, 23, 29, 31, 37) add nothing more than chaotic wind-bells effect. The resulting patterns are way too long to be digested by the regular brain, but who knows, maybe some savant would do. Anyway, quite a good accompaniment to get insane :)

Come get SoME!