Showing posts with label gtest. Show all posts
Showing posts with label gtest. Show all posts

Saturday, 21 February 2015

C++ BDD with Igloo/Bandit/Homebrew


It's been a while since I posted, so I thought it would be good to give an update.

We had some problems working with SpecFlow using C++, mainly around the "shim" layer we created to provide interopability.

The major issues were :-

1. Debugging ("the breakpoint game" trying to debug across layers)
2. C++ developers are not C# developers, and not all of them want to be
3. Management perception of reliance on C# as a risk
4. Fragile shim layer that doesn't allow the transport of all data types/classes

There were also a plethora of "niggles" which we mostly worked out, but overall no one was particularly happy with the solution.

Instead we decided to try "Igloo", a C++ BDD framework one of our developers found (we layer upgraded this to "Bandit"). It essentially provides macros to facilitate human readable specification tests :-
http://banditcpp.org/

We augmented this and developed Python scripts that allow us to translate a feature file something like this :-

 

   Feature: EatingApple

   Scenario: First bite


    Given an apple waiting to be eaten
    When teeth are sunk into it
    Then juice flies around it



Into this :-
Feature("EatingApple") Handler(EatingAppleSteps)
 
   GTestID(first_bite, EatingApple)
   Scenario("First bite")
      Given("an apple waiting to be eaten") StepPlay(GivenAnAppleWaitingToBeEaten)
      When("teeth are sunk into it") StepPlay(WhenTeethAreSunkIntoIt)
      Then("juice flies around it") StepPlay(ThenJuiceFliesAroundIt)
   EndScenario

At this stage we are now looking at directly executable C++ code. Cool, right? It uses something rather magical called a StepHandler class. As you can probably see, it manages the parsing of parameters in the human-readable feature file (I've left them out of the above example for simplicity) :-
class StepHandler
{
public:
   std::string GetCurrentLine();
public:
   void SetCurrentLine(std::string line);
   void Echo();
   void NotImplemented();
   int get_CurrentRow();
   void set_CurrentRow(int currentRow);
   void IncrementRow();
protected:
   std::vector<std::string> m_parameters;
   std::string m_currentLine;
   int m_notImplemented = 0;
   void ClearParameters();
   void ExtractParameters();
   void StripDelimiters(std::string& parameter);
   std::string GetStringParameter(int index);
};

The outputted C++ code's Handler() macro indicates which step handler the StepPlay() macros use to make their calls :-
class EatingAppleSteps : public StepHandler
{
public:
   EatingAppleSteps();
   ~EatingAppleSteps();
   void GivenAnAppleWaitingToBeEaten();
   void WhenTeethAreSunkIntoIt();
   void ThenJuiceFliesAroundIt();
};

In case you hadn't already realised, the constructor and destructor constitute the setup and tear down for each of the tests.

Beautiful, right? We now have a full C++ BDD system with many supporting scripts, fully integrated with google test, and are very happy programmers again.

Credit to James for his amazing work on this system and Seb Rose for the warnings which helped us avoid several dangerous pitfalls on the way - primarily the importance of synchronised feature/executable feature files. We generate an error in the build if they don't match. This was an excellent steer.

Feel free to contact me if you have any questions :)

Sunday, 15 June 2014

BDD of Unmanaged C++ using Visual Studio 2013 and Specflow

As part of a great BDD training session by Seb Rose, we went about setting up an executable specification environment for our existing C++ codebase. I'm going to talk very briefly about what this is for, and then go into some technical detail about some of the problems we had, and how they've been resolved.


A bit about BDD
BDD - or Behavioural Driven Development - is a method by which you convert non-technical, "behavioural" requirements into executable code, which can then be used to verify the behaviour has been implemented.

Through a process of deliberation with a product owner, tester and technical person, you derive stories, rules, examples and scenarios in a language understood by everyone, which then becomes an executable specification, which you can use to verify behaviour of the system.


Specflow, Cucumber and Gherkin
The scenarios are parsed. If you're uisng Specflow, the language used is Gherkin, which is parsed by Cucumber. You can read all about here :-
https://github.com/cucumber/cucumber/wiki/Gherkin

The output of this process is C# "feature" stub code for all the Given, When and Then statements found with a feature, with some regular expression powered attribute tags to parse the scenario text into function parameters.

This is cool.


Our setup
We were setting up Specflow in Visual Studio 2013 for use with C++ code that is cross-compiled in IAR Embedded for ARM chips. We do all our unit testing in Visual Studio because frankly, IAR Embedded is terrible. Our test target code is therefore unmanaged C++ code.

This introduces some significant complexities.

We created a fresh C# MSTest project within our existing unit test solution. I've called this "SpecificationTests" in its latest incarnation. We then added Specflow to the plugins for this project (right click the project --> manage NuGet packages).

C# can't talk directly to our unmanaged C++, so we had to create a "shim" layer in C++/CLR.

I was hung up for a while on a good way to organise this underyling "step test code", and Seb suggested by domain entity, so I created a "ScreenSteps" unmanaged C++ project, linked to our REAL code, and a "ScreenStepsShim" C++/CLR project linked to the ScreenSteps project and then added ScreenStepsShim as a reference to the SpecificationTests C#/Specflow project. The "domain entity" in question here, in case it isn't obvious, is the Screen!


The Problems We Had And Their Resolutions 

Specflow extension needs to be added to Visual Studio
In addition to adding the plugin to your project, you need to go Tools / Extensions and Updates and search in the online section for SpecFlow, then add that. This was the easiest problem :)


Code generation

Runtime library needs to be the same for projects within the solution.

"Inherit from defaults" seemed to disappear and reappear as an option at will in 2013, so forget about that. 

I set everything to multi-threaded debug DLL (Project properties / C++ Code Generation / Runtime Library). I also added the google test project we're using for unit testing into the solution so I could re-build that rather than hard-linking to it.


DLL linkage

Because the unmanaged DLL wasn't exporting the functions, a .lib wasn't being generated and the shim couldn't link to the steps DLL underneath it.

What was missing is DLL export code in the unmanaged C++ header :-


#ifdef SCREENSTEPS_EXPORTS
#define SCREENSTEPS_EXPORT __declspec(dllexport)
#else
#define SCREENSTEPS_EXPORT __declspec(dllimport)
#endif

class SCREENSTEPS_EXPORT ScreenSteps
{
   ...
};

Now there are functions to be exported, so a .lib is generated (otherwise the linker assumes it is pointless, because there are no functions to call).


Unmanaged DLL not copied by default

AKA "exception thrown by target of the invocation could not load file or assembly or one of its dependencies"...

This was a fun one. When you start trying to call your unmanaged code, it won't work unless the library has been copied into the folder the specflow tests are executing in. The managed library gets copied no problem, but for some reason the unmanaged one does not.
 

Then, when you try and set the "copy local" option to true, it resets itself to false!!! Wow.

This is apparently a common problem for Microsoft. See here :-
https://connect.microsoft.com/VisualStudio/feedback/details/766064/visual-studio-2012-copy-local-cannot-be-set-via-the-property-pages

They fixed it for a while, and it broke again! So now you have to do it manually in the project file!! Wow.

 Adding "<CopyLocal>true</CopyLocal>" to the vcxproj file for ScreenStepsShim seems to work.

<ProjectReference Include="..\ScreenSteps\ScreenSteps.vcxproj">
      <Project>...</Project>
      <CopyLocal>true</CopyLocal>
</ProjectReference>


Test Explorer hang during test run
This is really annoying. It doesn't happen if you just re-run the tests after changing the feature file, but does happen if you change the code THEN re-run the tests.

When you re-start Visual Studio, you can re-run the tests!

The problem is caused by vstest.exectionengine.x86.exe... I proved this by killing the process and re-trying without restarting Visual Studio. It seems to get excited when you re-build the other libraries and prevents completion of the specflow test build.

Finally I found this :-
http://stackoverflow.com/questions/13497168/vstest-executionengine-x86-exe-not-closing

Add a pre-build event :-

for 64-bit:
taskkill /F /IM vstest.executionengine.exe /FI "MEMUSAGE gt 1"

or for 32-bit:
taskkill /F /IM vstest.executionengine.x86.exe /FI "MEMUSAGE gt 1"

This entirely solves the test runner problem.


std::string to String^ conversion
Unmanaged string to managed C++ String^ (which C# can interpret into a C# string) requires a gcnew in the C++/CLR project, like this :-

      string original = someUnmanagedObject.get_SomeText();
      String ^converted = gcnew String(original.c_str());



Hopefully this saves other people some pain.