Automate The Boring Stuff With Python Review

I was pretty bored over the weekend and found this book on special. Normally I would not buy it since it’s targeted at the beginner level but I got to read for free.

It was a nice surprise and I would highly recommend it to anyone starting into the programming world. If you are planning on using python the examples in the book are well described and straightforward to do. I felt the sense of power that computing has and a reminder that programming can actually make day to day tasks better. My favorite example in the book was working with Pdf’s to add a watermark. Just for the fun of it I combined with the email sample to make an app that takes a folder of pictures adds it into  a pdf watermarkes them and email them.

Travis CI + Windows Azure App

For one of my open source projects(debesalgo.com) I ended up setting travis CI to build a windows azure application (asp.net) and I wanted to record my findings since some of the problems are not very descriptive.

Build

To start you go to travis-ci.com and make and account then you have to basically mark your project active and submit a .travis.yml file to get it running.

I suggest you start with this:
language: csharp
solution: debesalgo.sln
mono:
– latest

Once you save that into your file your build will attempt to run and you will see and error about tools not matching and the version not being supported this happens if you are using VS2015 and anything above framework 4.5 So let’s downgrade our application to 4.5(Note further released of xbuild will solve this).

Now the next error will mention System.Web.Entity for some reason this file does not download from Nuget so add it to you bin and make sure you force commit it into git.

Once that step is done they build will kick once more and you will be at the point that it says there is an error with your pdb go into your project settings and compile your release version without pdb’s (Note you can use EmitOptions to do a portable pdb also) that will clear this error and get the build phase of this project done. It should look green and building.

Unit Test

Go into your test project remove the reference to the Visual studio quality tools since those are not supported in mono. Add the nuget reference nunit (I’m using 3.0.1) and to the Nunit.VisualStudio.TestAdapter package. Once that is done go and make a test case that we know will always pass. I tend to do this:

[TestFixture]
public class SampleTest
{
[Test]
public void Empty()
{
//Sample test for those joining the project.
Assert.Pass();
}
}

Once we got this we are ready to go modify our travis file to run our unit test we will do this by scripting so it downloads the runner console and additionally makes a call to the compile dll.

My final code for the .travis.yml looks like this:

language: csharp
solution: debesalgo.sln
mono:
– latest
install:
– nuget restore debesalgo.sln
– nuget install NUnit.Runners -Version 3.0.1 -OutputDirectory testrunner
script:
– xbuild /p:Configuration=Release debesalgo.sln
– mono ./testrunner/NUnit.Console.3.0.1/tools/nunit3-console.exe ./debesalgo.Tests/bin/Release/debesalgo.Tests.dll

This configuration assumes a new asp.net azure project but it works and it will compile. If you want to add test code coverage SharpCover is a great tool to use on this stack the big difference is that you will need to use a debug build.

Thanks for reading and remember to add your build badge to your page.

So you want to create a UI System

So I had to do a research project last year I decided to do a UI system for Azul UI Depaul’s internal framework. Learned a lot about doing 2d on top of 3d and the different libraries available. Below my notes in case anyone wants to compare. At the end there are two documents attached that show usage and general design.

Introduction

The purpose of this document is to outline the process that got us to the result that we have today with PaletaUI we will highlight the problems and the solutions that were picked. We will also conclude on lessons that we go thru out the process. This differs from the Design Document which talks about what actually got constructed and from the usage guidelines which are meant for the end user if we have one for PaletaUI.

Topics

How to change azul to ortographic mode.

This was the first problem that we faced. In Azul or better said my custom version of azul we were not using the traditional camera but we had a system that involved our custom camera and a camera manager. We wanted to keep that functionality and still be able to render In 2d so what I decided was that in the rendering loop we were going to split the two parts so this problem became simpler to just how we set the camera. I implemented a method for orthogonal projection to pass to the shader manager in order to be able to handle this. The method is static in camera manager is called GetOrthogonal.

How to draw on top of a 3d scene

Like we mentioned above we had to split the rendering cycle into 2d and 3d. I wanted to isolate Paleta as much as possible from Azul. So I created a class called UIManager to be the external facing point for the drawing call. Then in render scene I call ui manager update and draw after the scene from Azul does its call. We decided that the 2d equivalent to the Scene was the View and that we were going to Have only one active at a time. Additionally the Scene was going to assign the view using UIManager and be responsible of filling it.

How to draw a quad

We have our call set and know we need to be able to make a square because the first object we were going to create was basically a panel. We used the same rendering technique used in the rest of Azul create a predefined Vao. The only difference is that we are not using a translation function but recreating the quad if it ever moves. This is due to the fact that on the 2d ui things hardly ever move.

How to Control Rendering Order

Basically drawing on the screen is a stack so we needed a way to order ourselves. The decision that was made was to add the frame basically and object that only purpose is to be read in the order it should be drawn so a view has many frames.

Different types of Settings needed for the quads

To be able to display Text , Backgrounds and images I had to use control flags to indicate when to load a texture and load the texture shader, the specific text character shader or simply draw solid. I handled inside the quad class.

How to load fonts

To load fonts we used the library freetype which allowed to grab fonts directly out of ttfs. All fonts load from the same folder the application is running from. Note in the visual studio solution is the main folder for the solution.

How to Store the Font

Decided to make a font manager that basically contains a hashmap of the loaded fonts one font basically means a font file in a specific size. We basically create a texture for every character.

When to Store the Font and for how long

We load them until someone tells us to unload. They are fairly expensive to read and turn into textures in memory so we are super lazy on letting them go this is handled on the FontManager Class

How to calculate position for the letters

This was pretty tricky we used a technique extremely similar to the one learnopengl.com We basically create a character struct to give us width height and the bearing so we can do proper spacing between characters. The logic to do this is inside label since is the class that everyone else that has text inherits.

What Shaders to use with Freetype

We had to create our own fragment and vertex shader. Basically what our shader does is that it looks at the image and uses the color red from the texture to set the alpha. The textures from freetype have Red characters with black backgrounds. Additionally it substitute any color that you send into it to replace the red so we can print in different colors.

What Structure should we use for the UI objects

Once we had figured out how to render text we were ready to actually start working on creating a compelling structure to place things on the screen. Added to the View and Frame decisions that had been previously made. I decided on the widget to be the base class for all representing a rectangular area of the screen and made the coordinate system to work from the bottom left. So we have two functions to basically push from the bottom and push left a widget.

How do we keep our structure open

It was my goal to allow people to create their own components later and to be able to make aggregated components like the highscore one in the demo. Due to this the widget class is an abstract that allows to be inherit and can redefine its draw and update so that it can be manipulated.

Is Panel a special type of widget

Not really but yes the difference with panel is that it works like an anchor basically allows you to add many widgets to it and anchor them to its x and y.

Are Buttons and Textbox special kinds of labels

It made sense for buttons to inherit from labels because they needed to show text and a background it ended up working very well. I also thought that textbox was a button and then realize that it was not since it only shared one characteristic that it could be clicked.

How do we take mouse input without getting too attached

To solve this problem I created a Virtual Mouse that needs to be updated every cycle. It works well and reduces all the data to normal ints.

How do we take a click

I did not made an elaborated collision system. Realized that you only have 5 o6 buttons or textboxes per screen decided to let them manage it themselves by doing a selfcheck on update. This is a possible area for improvement later on.

How do we take the actions that a button needs

Created a command system for this you need to inherit the class and create a specific command for your action. It ended up being great it encapsulates the action and makes them reusable. There are two commands implemented natively in the system on debug prints that something got clicked. The second one set focus to the widget that uses it.

How do we load the images for our system

I wanted to use Soil and add an image loader similar to the font manager but due to problems with SOIL ended up using the tga loader and texture manager from azul. So for now they need to be added to the enum. This is my number one area for improvement.

How do we take keyboard input without getting too attached

The same solution that we used for mouse would not work here. What we ended with was creating a keystroke observer that only has one subject which is whatever widget that is focused at the moment. Which created the problem of having to track and reset focus.

How do we keep track of Focus

We use focus to know where to send our keystrokes right now for the textboxes. There is a FocusManager class that gets set via the SetFocus Command Whenever a textbox is click other classes can use this command to set focus to themselves if they need it. The focus is no hierarchichal only one widget at a time.

What format should we use for declarations

I started on the route of using protocol buffers and thought it was a good idea had to do a reversal and fallback to xml for the following reasons. The compiler adding it to the solution was hard and while I solved this and got it running in the solution was worried that my setup would fail later. Second Reason and I think is the most important is that custom parsing of hierarchichal objects was really complicated. Seems like protocol buffers was meant to handle flat objects which makes sense for messages instead of nested hierarchy’s. With that said the format is pretty need and I think we should try to use it to create a duel networking system with protocol buffer serializing the messages. Is one of the projects I’m suggesting for the next semester.

I decided to use tinixml2 class to do deserialization.

How do we load the declarative format

Added a Load method on the View class and the frame class and that solved it for the top level objects. I only had to give a filepath to the view to be able to load an entire structure. With that said this created the problem that we have to be able to find the objects and ended up having to make the hasname interface that lets me search thru the frames and widgets.

How do we deserialize nested objects

This was an interesting problem and it was around the widgets ended implementing a singleton class widgetserializer which host a list widgetserializercomponents and uses a chain of command pattern to figure out who it matches with and executes it for that specific section.

How do we keep our serialization logic apart from our objects

We did not include the logic at all in the widgets but made a parallel structure in charge of loading them up.

How do we extend the serialization for custom components.

Widgetserializer has and add method to let you register widgetserializercomponents that were not created has part of the framework.

Lessons Learned

Libraries can take a while to integrate

If you abstract well you can forget about the underlying presentation

Let the user control lifetime of loaded objects

Deserialization != Construction

A few Ui Pieces can make an app a lot better looking

Abstractions for input can be complicated

Using Commands for Events is a good way of encapsulating logic

The Ui View is a separate element controlled by the scene

Improvements

Better image loading

Clipping to size.

Add Border to Widget

Add Gradients to background Circular and linear.

Parse Colors defined by rgba in the declaration xml.

References

1. http://voxels.blogspot.com/2015/07/opengl-game-gui-widgets-with-source.html

2. http://openglgui.sourceforge.net/

3. http://www.gamedev.net/topic/610816-how-to-make-my-own-gui-using-opengl-and-sdl-and-c-or-whateva/

4. http://learnopengl.com/#!Getting-started/Hello-Triangle

5. https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=17&cad=rja&uact=8&ved=0CEkQFjAGOApqFQoTCL_r4b6up8cCFUZXPgodO60Ctg&url=http%3A%2F%2Fwww.uwplatt.edu%2Ffiles%2Fcsse%2Fcourses%2Fprev%2Fcsse411-materials%2Ff13%2Ffeldmeiera-gui%2520_programming.doc&ei=djzNVb_YI8au-QG72oqwCw&usg=AFQjCNEj0gnQbaiDdux75ToDv_RUtmfiog&sig2=uj47NywhsjTEZNaakMQjNQ&bvm=bv.99804247,d.cWw

6. http://www.codeproject.com/Articles/30775/TetroGL-An-OpenGL-Game-Tutorial-in-Cplusplus-for-W

7. https://developers.google.com/protocol-buffers/?hl=en

8. http://www.grinninglizard.com/tinyxml2docs/

9. OPEN GL SUPER BIBLE

Videos

Milestone 1- https://www.youtube.com/watch?v=9jAB_ZPbuyc

Final – https://www.youtube.com/watch?v=m42b73GEnrw&feature=youtu.be

Link to Design Document

https://www.scribd.com/doc/296137070/PaleTa-UIi-Design-Documentation

Link to usage guide

https://www.scribd.com/doc/296137071/PaleTa-ui-Documentation

BookMArkLet vS Chrome plugins

I spend quite a bit of time researching bookmarklets and specific browser plugins in particular chrome plugins. Below my notes on how to get started and what my conclusions where of when to use the different technologies.

Bookmarklets

Let’s start first with the bookmarklet’s since they are the older technology. Basically is an anchor that runs javascript and allows you to execute javascript on the page it is. If you bookmark this anchor it means that you can have a piece of javascript on whatever page the user is browsing which is really useful if you want to do dom sensitive operations. Like for example reacting to a specific page and provide an ui or adding a button to the netflix web interface(someone did that).  Let’s look at a small example if we add this to a link it will let us create a quick target link for geni.us the place I work.

javascript:location.href=’http://target.georiot.com/Proxy.ashx?GR_URL=’+encodeURIComponent(location.href)

Click me

You are going to end in the bad link page because my blog is not something valid to process. With that said if you go to amazon.com you will see it process thru.

For Bookmarlets over the years the capabilities for have increased with the ability to add libraries and your custom javascript files into the bookmarklet. Additionally they currently got support on every browser on the desktop and mobile(at least the big 3’s). So in conclusion they are great for:

  • Manipulating the dom.
  • Adding new elements to a page.
  • Cross Browser compatibility.

Below a few links that help. I really recommend using the bookmarlet generator since it trivializes the process of creating a bookmarlet.

Bookmarklet Basics

Bookmarklet Generator

Chrome Plugins

Alright So all the browsers now a day have a plugin system that is proprietary and work by using javascript to call api’s they have exposed in a sandbox. 

The plugins can do all that a bookmarlet can except be cross browser compatible. Additionally to that they provide a few advantages in the case of chrome:

  • Access to the chrome api’s – This allows you to store data, set alarms even capture the desktop has an image. Aditionally they include new api’s so if you need any of these capabilities doing a plugin is a great idea. Documentation
  • They give you access to the store. Bookmarklet’s need to be discovered via someone hitting your site. The plugins by being on the store you can benefit from someone browsing the new arrivals list.
  • You don’t have to click to activate it. Is running on the background so if your experience would benefit from not having a click to start this is the way to go.

In terms of effort required for simple task they are both very straight forward and there is clear documentation on the web.

With that said the not being cross-browser compatible is a really big weakness in terms of maintenance.

How to build a chrome plugin

Aussie Puncharoo is out

After a lot of toil and fun the game I have been creating with the talented Louis Lonsway is finally out is the story about and australian bloke and his quest to get a Pint of Beer. Like everyday in the outback he gets attacked by tons of wild animals.

The game started has part of ludum dare 20 and me wanting to learn how to build a platformer. http://alfredoalvarez.com/blog/?p=160

To be honest I made a very big mistake building this game which was to focus on building and engine and spent less time on the game. Right now the game supports a lot of scripted mechanics not all which made it into the game.

The game is now on the store which is awesome and I can see that the process from the windows developer dashboard has been very streamlined.http://www.windowsphone.com/en-us/store/app/aussiepuncharoo/0cd99df2-fb9e-4774-8169-a9053f32c4ff

While I celebrate the victory of shipping I have already found the first list of improvements and bugs including a grave in which the game seems to be crashing to people with phones from the seven series. To the end of the development phase and the beginning of the support one.

Getting back on the Selenium horse

For those that don’t know i left my position in Microsoft to go work for the startup that powers www.georiot.com. Because of this i have gotten in touch with a lot of tools that i had not used for awhile.

This post is my experience of coming back to Selenium from a 6 month Hiatus. This time i used it with the C# webdrivers library and ran it using the firefox browser.

The Good

  • I feel like the API has improved and getting a simple prototype of my automation took a manner of minutes.
  • The firefox profile class is completely straightforward.
  • The support for ghostjs and the possibility of running headless is pretty awesome.
  • Multiplatform support.

The Bad

  • C# drivers in Mono are kind of finicky(repo linked on the bottom fixes the zip dependency by moving from Ionic to Sharp Zip).
  • The User group in google groups does not get has many responses has i hope.

To talk a bit about my task i wanted to login in to a website an click a link to download a file. This was trivial after figuring the firefox profile. The profile below does the following configures Firefox to save & download files to a certain place without asking for confirmation.

var profile = new FirefoxProfile();
profile.SetPreference(“browser.download.folderList”, 2); // This item tells firefox  to use a specific directory instead of default
profile.SetPreference(“browser.download.dir”, DownloadFolder); //The specific directory to download
profile.SetPreference(“browser.helperApps.alwaysAsk.force”, false); // Tells firefox to not ask what you want to do with a file
profile.SetPreference(“browser.download.manager.showWhenStarting”, false); // Indicates that the download window should be off
profile.SetPreference(“browser.helperApps.neverAsk.saveToDisk”,
“text/xml, text/csv, text/plain, text/log, application/zlib, application/x-gzip, application/x-compressed, application/x-gtar, multipart/x-gzip, application/tgz, application/gnutar, application/x-tar, application/gzip”); // specifies all the extensions that should be auto download.

Definition of this items can be found at http://kb.mozillazine.org/About:config_entries

Thanks to free_at_last from reddit for helping me improve the article and the concise line to summarize the code.

My problems came when i started trying to run on mono using firefox at first like i mentioned i got an error that said path not found that i traced to this issue. Basically the library does not handle the paths in linux.

https://code.google.com/p/selenium/issues/detail?id=3804

I resolved that issue by switching the zipping library and placed the code for it here:

https://github.com/aallfredo/selenium

After that firefox would open but then i got into the socket error problem tried installing and older version of firefox and posting into the user group but no solution. (Note also tried it with the ghostjs drivers and they really don’t run in linux with mono). For the meanwhile im going to use a windows box to get me around.

 

So in summary great new API mono supports seems to be a bit lacking but is an open source project so it could be me that takes the mantle to resolve it on that.

 

Validating if a Sudoku solution is valid

I got this problem today has part of a conversation that I was having and thought I was interesting. Basically how can you in O( n ) validate that a 9×9 soduku is valid.

There are 3 rules that need to be followed for a validation to return true:

  1. For each row there can only be and instance of each numbers.
  2. For each column there can only be and instance of each numbers.
  3. For each cluster of 9 there can only be and instance of the set of numbers.

The trivial solution that comes to mine is to go and count for each of this different conditions but that would require at least 3 passes tru the grid. For my solution i decided to just store a hashset for each of the conditions that need to be valid and stop on a negative. We use the numbers has distinct markers and not has numbers in particular since being number has no properties that help us. Solution is below:

using System.Collections.Generic;

namespace Algorithms
{
public class SudokuPuzzleValidator
{
readonly int[,] _board;

 

public SudokuPuzzleValidator():this(newint[9, 9])

{}

 

public SudokuPuzzleValidator(int[,] board)

{

_board = board;

}

public bool Validate()
{
const int integersInGame = 9;
var rowSet = new HashSet<int>[integersInGame];
InitializeSet(integersInGame, rowSet);
var columnSet = new HashSet<int>[integersInGame];
InitializeSet(integersInGame, columnSet);
var subGridSet = new HashSet<int>[integersInGame];
InitializeSet(integersInGame, subGridSet);

for (var row = 0; row < integersInGame; row++)
{
for (var column = 0; column < integersInGame; column++)
{
var cval = _board[row, column];
if (rowSet[row].Contains(cval))
{
return false;
}
rowSet[row].Add(cval);
if (columnSet[column].Contains(cval))
{
return false;
}
columnSet[column].Add(cval);
var subGridNumber = FigureOutSubGrid(row, column);
if (subGridSet[subGridNumber].Contains(cval))
{
return false;
}
subGridSet[subGridNumber].Add(cval);
}
}
return true;
}

private static void InitializeSet(int integersInGame, HashSet<int>[] rowSet)
{
for (var i = 0; i < integersInGame; i++)
{
rowSet[i] = new HashSet<int>();
}
}

private static int FigureOutSubGrid(int row, int column)
{
return column/3 + row/3*3;
}
}
}

Finding All empty tables in SQL db

Had to do this at work figured this script might be useful to some around here. If you are and tester and gets a new system is a good way to find out what tables are getting no activity.

DECLARE @Tables TABLE
(
Name nvarchar(100)
)
DECLARE @EmptyTables TABLE
(
Name nvarchar(100)
)
DECLARE @AlmostNoRowsTables TABLE
(
Name nvarchar(100)
)

INSERT INTO @Tables
SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.TABLES

DECLARE @controlVariable nvarchar(100)
DECLARE @counter int
DECLARE @rowsleft int
SELECT Top 1 @controlVariable = Name FROM @Tables

Declare @SQL nVarChar(1000)
Declare @ex nVarChar(1000)
SELECT @SQL = ‘SELECT @counter=COUNT(*) FROM ‘

WHILE(@rowsleft > 0)
BEGIN
SELECT @ex = @SQL + @controlVariable

EXEC SP_EXECUTESQL @ex,N’@counter INT OUTPUT’,@counter OUTPUT

if(@counter = 0)
BEGIN
SELECT ‘Empty Table’ +@controlVariable
END

if(@counter > 0 AND @counter <10)
BEGIN
SELECT ‘Table has almost no rows’ +@controlVariable
END
DELETE @Tables WHERE Name = @controlVariable
SELECT Top 1 @controlVariable = Name FROM @Tables
SELECT @rowsleft = COUNT(*) FROM @Tables
PRINT @rowsleft
END

Learning a new mocking framework what you should learn (1 out 3)

Over the past six month depending on the workplace that I went I had to unit test. Most of the places where using Dynamic mock frameworks. After learning one or two I came up with a  list of what one should learn from a framework to make life easier.

What you need to learn of a mocking framework

·        -Return  values for interfaces

·        How to fire events

·        How to mock properties set and get

·        How to register params

·        How to ignore args

·        Work with generics

·        Work without parameters

·        Register handlers

·        Work with out parameters

·        Have a mock implement two interfaces

In the next article I will use a moq framework to show examples of each.

I survived burnout just barely

Hi blog readers I have recently read a bunch of posts regarding Work life balance. Starting with people trying to achieve it:

http://501manifesto.org/

http://www.hmans.io/2012/04/23/the-501-developer-manifesto/

Followed with some tips to survive the burnout and detect it:

http://www.codingninja.co.uk/logout-before-you-burnout/

And quite a few who don’t believe is an issue an most of us talking about this are a bunch of whiners.

I’m just going to state this I did not believe in burnout, it happened and I almost did not made out of it. I see there are two types of engineers in big companies those that make programming their lifestyle and those that just see it has a skill they have for work and two earn money. To the second kind I really can’t tell you much about burn out since except for the last 4 months I have never behave that way.

To the first kind listen if you want even though has you read it its probably going to sounds has rubbish since you know better Smile. You are not going to see it coming unless someone lets you know. Second its not about the time you spend in the office but about the time you are in the office but the time you spend connected to the problems that you need to solve. Especially those that drain your energy (In my case solving political or build issues).

So how does burnout feel. You know when you have those days that you look at the screen and no code wants to come out and your mind is like lets go look at reddit for a few more hours is like that but 24/7 and your mind seems to want to stay consuming instead of producing.

So how do you get to that point. First and foremost its your fault. In my personal case I was having confidence issues because I had tried to move to another team in the company and got rejected and decided to try to improve myself. If you feel at work that you have something to prove that is one of the early signs. Second things start getting overcomplicated and you can’t really justify it therefore you try to fix (this is normally not a real problem talk to your boss). Third you feel like you need to be in this job since getting out would be admitting failure at this point I would say that you know that you are in trouble. 4th very close to the end I felt like everyone was doing something wrong around me(Normally I never think that).

So what happened next I stopped functioning got to the point that I could not do anything and decided I had to quit.

So how did I get back on my feet quite a few things:

  • Family and gf support -Can’t say how important this was
  • Changing my lifestyle to focus on financial security
  • Watching tv for 3 whole weeks.
  • Reconsidering my career choices and if being a developer was a good choice.
  • Last but not least getting to work in a place where i was able to work in the part that i enjoyed of the job without all the other ones(This is the one that is hard to find).

    If you are wondering what im doing forward to avoid a repeat I will tackle that in a following post later this week since its rather lenghty. The one thing I would like to mention is that is all in your control burnout has nothing to do with bosses or what not in all honesty people should not have much power over you if they do. The first thing that you need to do is fix your living situation to help avoid burnout. Hopefully this experience helps some people out there. If there is anything you guys want to discuss let me know.