nnCONDUCTOR

In order to get a good assessment of the abilities of the nn-os a simple application was developed for a Neural Network conductor that learns from the user gestures of one hand, note that all code below runs on the BeagleBone board super fast:

1. Movements: Movement to the right (+0.5), movement to the left (-0.5), stand still movement (0.0)
2. 30 notes of Beethoven's Ode To Joy were chosen for music, in frequency domain, could have been in full MIDI but for now a simple set of frequencies to a sinusodial sound generator would suffice:


/* define the notes by the frequencey of their sinusodial */
	c4 = 261.64;
	d4 = 293.68;
	e4 = 329.64;
	f4 = 349.24;
	g4 = 392.00;
	a4 = 440.00;
	b4 = 493.92;
	

/* TODO: this should come from the MIDI file */
	OdeToJoy[0] = e4;
	OdeToJoy[1] = e4;
	OdeToJoy[2] = f4;
	OdeToJoy[3] = g4;
	OdeToJoy[4] = g4;
	OdeToJoy[5] = f4;
	OdeToJoy[6] = e4;
	OdeToJoy[7] = d4;
	OdeToJoy[8] = c4;
	OdeToJoy[9] = c4;
	OdeToJoy[10] = d4;
	OdeToJoy[11] = e4;
	OdeToJoy[12] = e4;
	OdeToJoy[13] = d4;
	OdeToJoy[14] = d4;
	OdeToJoy[15] = e4;
	OdeToJoy[16] = e4;
	OdeToJoy[17] = f4;
	OdeToJoy[18] = g4;
	OdeToJoy[19] = g4;
	OdeToJoy[20] = f4;
	OdeToJoy[21] = e4;
	OdeToJoy[22] = d4;
	OdeToJoy[23] = c4;
	OdeToJoy[24] = d4;
	OdeToJoy[25] = e4;
	OdeToJoy[26] = e4;
	OdeToJoy[27] = d4;
	OdeToJoy[28] = c4;
	OdeToJoy[29] = c4;

3. Based upon the change in the notes a perfect conductor is calculated

REAL nnSign (REAL a)
{
	
	if (a > 0.0)	return 0.5;
	if (a < 0.0)	return -0.5;
	
	return 0.0;
	
}




for (i = 0; i < Units[0] - 1; i++)
	{
		/* best theoretical conducting possible */
		input1[i] = nnSign (OdeToJoy[i+1]-OdeToJoy[i]);
		target1[i] = OdeToJoy[i]/b4;
	}

printf("Perfect theoretical conducting \n");
	for (i = 0; i < Units[0] - 1; i++)
		printf (" %f", input1[i]);
	
	printf("\n\n");
0.000000 0.500000 0.500000 0.000000 -0.500000 -0.500000 -0.500000 -0.500000 0.000000 0.500000 0.500000 0.000000 -0.500000 0.000000 0.500000 0.000000 0.500000 0.500000 0.000000 -0.500000 -0.500000 -0.500000 -0.500000 0.500000 0.500000 0.000000 -0.500000 -0.500000 0.000000

3. NN created to handle the backpropagated adaptive learning


nnInit (&net);
	
	
	NUM_LAYERS = 3;
	
	/* dynamic array allocation */
	Units = (INT *) malloc (NUM_LAYERS * sizeof(INT));
	
	/* 30 inputs, 64 hidden middle layer, 30 outputs, total 3 layers */
	/* 5 for fingers of the hand, if I play them one after the other, what will
	 NN play in return to adjust to the OdeToJoy according to its adaptive learning
	 */
	Units[0] = 30;
	Units[1] = 64;
	Units[2] = 30;
	
	nnGenerateNetwork(&net, NUM_LAYERS, Units);
	nnRandomWeights(&net, -1.0/1.0, 1.0/1.0);
	

Let's see how accurately this NN does learn the perfect theoretical hand-motion to Ode To Joy notes, and the learning was repeated 300 times which is an overkill but wanted to test and see how fast the Beaglebone vertorization works:

for (i = 0; i < 300; i++)
	{
		nnSetInput(&net, input1);
		nnSimulateNet(&net,input1, output1, target1, TRUE);
		
	}
	
	
	printf("Let's check the errors, the first row is the NN evaluation after training, the second row is what it was supposed to be: \n");
	
	printf("\n");
	for (i = 0; i < Units[2]; i++)
	{
		printf("%f ", b4*output1[i]);
	}
	printf("\n");
	printf ("Best theoretical training \n");
	for (i = 0; i < Units[2]; i++)
	{
		printf("%f ", b4*target1[i]);
	}

The first row of numbers are the NN Eval which is taking as input the perfect theoretical hand-motions and generates the notes for Ode To Joy, see how close the frequencies are:


I am nnCONDUCTOR
Let's check the errors, the first row is the NN evaluation after training, the second row is what it was supposed to be:


329.632723 329.653514 349.236756 392.004078 392.032540 349.250540 329.634919 293.677107 261.635577 261.626550 293.681664 329.636790 329.630225 293.677600 293.689046 329.631732 329.639090 349.229227 392.001614 392.010019 349.252521 329.632836 293.673893 261.643927 293.701046 329.625913 329.638479 293.677933 261.630258 6.297173

Best theoretical training
329.640000 329.640000 349.240000 392.000000 392.000000 349.240000 329.640000 293.680000 261.640000 261.640000 293.680000 329.640000 329.640000 293.680000 293.680000 329.640000 329.640000 349.240000 392.000000 392.000000 349.240000 329.640000 293.680000 261.640000 293.680000 329.640000 329.640000 293.680000 261.640000 0.000000

Now let's mock some data as if the conductor was imperfect, of course:

/* TODO: this should come from the MIDI device by trials from actual conducting */
	for (i = 0; i < 30 - 1 ; i = i + 2)
	{
		OdeToJoyIMPROV[i] = 0.5;
		OdeToJoyIMPROV[i] = -0.5;
		
	}
	
	for (i = 0; i < Units[0] - 1; i++)
	{
		input2[i] = OdeToJoyIMPROV[i];
		
	}
	
	for (i = 0; i < 30 - 3 ; i = i + 4)
	{
		OdeToJoyIMPROV[i] = 0.5;
		OdeToJoyIMPROV[i+1] = -0.5;
		OdeToJoyIMPROV[i+2] = 0.0;
		OdeToJoyIMPROV[i+3] = -0.5;
		
	}
	
	for (i = 0; i < Units[0] - 1; i++)
	{
		input3[i] = OdeToJoyIMPROV[i];
		
	}

Now let's train the new hand gestures, 300 times in some order, again 300 is overkill:


printf ( "Let's train with the new genstures \n");
	
	
	for (i = 0; i < 300; i++)
	{
		nnSetInput(&net, input1);
		nnSimulateNet(&net,input1, output1, target1, TRUE);
		
		nnSetInput(&net, input2);
		nnSimulateNet(&net,input2, output3, target1, TRUE);
		
		nnSetInput(&net, input3);
		nnSimulateNet(&net,input3, output3, target1, TRUE);
		
		
	}

And NN Eval the input2 and input3 as the new conducting hand gestures,


/* Now use the NN to evaluate only set the training to FALSE */
	
	nnSetInput(&net, input2);
	nnSimulateNet(&net,input2, output2, target1, FALSE);
	
	
	printf ("IMPROV1 output \n");
	printf("\n");
	for (i = 0; i < Units[2]; i++)
	{
		printf("%f ", b4*output2[i]);
		
	}
	
	printf("\n");
	printf("\n");
	
	nnSetInput(&net, input3);
	nnSimulateNet(&net,input3, output3, target1, FALSE);
	
	printf ("IMPROV2 output \n");
	printf("\n");
	for (i = 0; i < Units[2]; i++)
	{
		printf("%f ", b4*output3[i]);
		
	}
	

The new gestures, though very different than the perfect theoretical conducting, are producing the same notes as in the original Ode To Joy:


Let's train with the new genstures
IMPROV1 output

329.640739 329.637722 349.240769 391.999369 391.993992 349.238244 329.641546 293.680524 261.641142 261.641275 293.679594 329.640366 329.641585 293.679726 293.678846 329.641542 329.638861 349.241386 391.998919 391.998110 349.238400 329.640911 293.680796 261.639473 293.677889 329.642396 329.639851 293.680071 261.641150 1.333444

IMPROV2 output

329.639012 329.651783 349.237853 391.995413 392.024284 349.245752 329.630880 293.674910 261.631967 261.636824 293.688278 329.640345 329.631024 293.685027 293.678352 329.635509 329.644757 349.233828 392.006185 392.014813 349.246216 329.638882 293.677430 261.643380 293.687260 329.630201 329.644626 293.680531 261.639583 4.099300


Let's make the same experiment but this time no training:


/* untrained input */
	
	for (i = 0; i < 30 - 3 ; i = i + 4)
	{
		OdeToJoyIMPROV[i] = 0.5;
		OdeToJoyIMPROV[i+1] = 0.0;
		OdeToJoyIMPROV[i+2] = 0.0;
		OdeToJoyIMPROV[i+3] = -0.5;
		
	}
	
	for (i = 0; i < Units[0] - 1; i++)
	{
		input4[i] = OdeToJoyIMPROV[i];
		
	}
	
	/* Now use the NN to evaluate only set the training to FALSE */
	
	nnSetInput(&net, input4);
	nnSimulateNet(&net,input4, output4, target1, FALSE);
	
	printf ("\n\nUNTRAINED IMPROV3 output \n");
	printf("\n");
	for (i = 0; i < Units[2]; i++)
	{
		printf("%f ", b4*output4[i]);
		
	}

Untrained output:


247.001183 324.423909 314.092314 453.881976 389.269045 242.455085 422.168809 351.463674 303.364334 255.216437 333.914283 221.099017 323.603458 283.520151 220.679470 382.151972 304.372105 353.983007 434.623649 333.220943 398.207716 294.526316 244.059593 276.314868 192.237255 385.651120 316.734008 323.127553 300.306695 5.860819


The sound of the untrained output, which can serve as an improvisation (some rapid noise in the beginning due to the export from Mathematica not in the data):

OdeToJoyIMPROVuntrained

Let's train this untrained data but just a little bit e.g. 4 times repeated learning:


/* nnSetInput not needed but placed in the code for the purpose of teaching how the code works */
nnSetInput(&net, input4);
nnSimulateNet(&net,input4, output4, target1, TRUE);
	
for (i = 0 ; i < 5; i++)
	nnSimulateNet(&net,input4, output4, target1, TRUE);
	
printf ("\n\nTrain for IMPROV3 output \n");
printf("\n");
for (i = 0; i < Units[2]; i++) 
		printf("%f ", b4*output4[i]);
		



Train for IMPROV3 output

335.832798 332.398557 351.158961 431.807949 393.989485 355.469383 335.469460 293.893473 261.059124 261.642879 291.563823 334.445296 330.962369 293.169041 296.538629 328.687413 331.194847 349.066182 406.753920 392.095134 350.793554 332.058776 296.847315 260.285059 297.137262 329.510238 331.725621 294.408333 261.199646 5.319829


The sound of the trained data, however due to the small number of repeated training there are slight off-beats:

OdeToJoyIMPROV

If we run the trainers more than 4 times the improvisation matches the original Ode To Joy!

This entry was posted in C/C++, Documentation, Electronic Music, Software, tutorial. Bookmark the permalink.

Leave a Reply