How to convert from pytorch to nnb for spresense?
-
Hi!
I want to train a NN in PyTorch and deploy the model on the spresense using the DNNRT library.
According to the NNabla docs there is a way to convert NNs from ONNX to NNB.
I tested this conversion and tried to deploy it on the spresense in the same fashion as in the DNNRT example. However it failed to initialize the runtime ("Runtime initialization failure") with errorcode -1.
To test if the conversion works properly, I exported a working model from the NNC as ONNX and tried the conversion with the following commands:
nnabla_cli convert -b 1 sony.onnx sony_nabla.nnp nnabla_cli convert -b 1 sony_nabla.nnp sony_nabla.nnb
However this ended with the same runtime initialisation error as before. From here I conclude that I am missing the correct conversion arguments for nnabla_cli, could you help me out with this?
Thanks a lot!
-
Hi @niciBume
Could you please try to find out where exactly
dnn_runtime_initialize
returns error?
https://github.com/sonydevworld/spresense/blob/87f81d9c35c7d05d6a75e9177b45d6cbe8d8c063/sdk/modules/dnnrt/src/runtime/runtime_nnabla.c#L66Without it, it's hard to say what exactly doesn't work because this error only says that something didn't work in the NNabla C-runtime library.
As for the conversion, I think you should use these commands. This is how it is described in the documentation:
$ nnabla_cli convert sony.onnx sony_nabla.nnp $ nnabla_cli convert -b 1 sony_nabla.nnp sony_nabla.nnb
Best Regards,
Kamil Tomaszewski -
Hi @kamiltomaszewski thanks for the reply!
The output of the dnnrt_lenet example is the following:
nsh> dnnrt_lenet --s /mnt/sd0/sony_nabla.nnb Load nnb file: /mnt/sd0/sony_nabla.nnb Load pgm image: /mnt/sd0/0.pgm Image Normalization (1.0/255.0): skipped start dnn_runtime_forward() up_hardfault: PANIC!!! Hard fault: 40000000 up_assert: Assertion failed at file:armv7-m/up_hardfault.c line: 148 task: sony_uav up_registerdump: R0: 0d0f6e54 0d09969c 0d099918 00000000 00000015 0d04b770 0d03d6b0 0d04b6e0 up_registerdump: R8: 00000003 00000000 00000000 00000000 00000009 0d04b6b0 0d024ae5 00000000 up_registerdump: xPSR: 00000000 BASEPRI: 000000e0 CONTROL: 00000000 up_registerdump: EXC_RETURN: ffffffe9 up_dumpstate: sp: 0d03aba8 up_dumpstate: IRQ stack: up_dumpstate: base: 0d03ac00 up_dumpstate: size: 00000800 up_dumpstate: used: 00000148 up_stackdump: 0d03aba0: 000007c4 0d00257b 000000e0 00000000 00000000 00000009 0d04b6b0 0d024ae5 up_stackdump: 0d03abc0: 00000000 0d03abd0 0d001a9f 00000003 00000000 0d001aa7 0d001a85 0d00354f up_stackdump: 0d03abe0: 000000e0 0d001a79 000000e0 0d04b5dc 0d04b770 0d03d6b0 0d04b6e0 0d000333 up_dumpstate: sp: 0d04b6b0 up_dumpstate: User stack: up_dumpstate: base: 0d04b720 up_dumpstate: size: 000007c4 up_dumpstate: used: 00000628 up_stackdump: 0d04b6a0: 00000000 80000000 80000010 0d024ae5 0d0f69a0 0d0f6900 0d04b770 0d020b7d up_stackdump: 0d04b6c0: 00015419 00000000 0d04ba00 0d03d6b0 00000001 0d020899 00000000 00000001 up_stackdump: 0d04b6e0: 0d03d6b0 0d04b770 00000249 00015419 00000000 00000000 00000000 00000101 up_stackdump: 0d04b700: 00000000 00000000 00000000 00000000 00000000 0d004377 00000000 00000000 up_taskdump: Idle Task: PID=0 Stack Used=0 of 0 up_taskdump: hpwork: PID=1 Stack Used=576 of 2028 up_taskdump: lpwork: PID=2 Stack Used=352 of 2028 up_taskdump: lpwork: PID=3 Stack Used=352 of 2028 up_taskdump: lpwork: PID=4 Stack Used=352 of 2028 up_taskdump: cxd56_pm_task: PID=6 Stack Used=352 of 996 up_taskdump: init: PID=7 Stack Used=1008 of 8172 up_taskdump: sony_uav: PID=16 Stack Used=1576 of 1988
This seems to be a nuttX error message, if I am not mistaking?
It actually passes
dnn_runtime_initialize
but fails atdnn_runtime_forward()
, where I trace it back to https://github.com/sonydevworld/spresense/blob/87f81d9c35c7d05d6a75e9177b45d6cbe8d8c063/sdk/modules/dnnrt/src/runtime/runtime_nnabla.c#L186Regarding the conversion, I tied your suggestion, yet with the same result. Would you know the build commands for the NNC?
Best,
Nic
-
Could you please try the converter from Nnabla version 1.12.0?
Best Regards,
Kamil Tomaszewski -
I downgraded Nnabla to 1.12.0 and it worked! Thanks a lot for the help!
Cheers,
Nic -
Hmmm... while the example (exported it directly from NNC as .onnx and .nnb to ensure that the model works) .onnx -> .nnp -> .nnb now works on the Spresense. However, with Nnabla 1.12.0 my actual PyTorch -> .onnx model can not be converted to .nnp anymore. This is a step that actually worked with Nnabla 1.20.0. I traced this issue back to Nnabla 1.14.0, where PyTorch -> .onnx works again, but the .onnx -> .nnp -> .nnb conversion yields the same error on the MCU as previously stated.
Is there a workaround for this?
Best,
Nic -
Forgot to add error print (and can't edit post anymore):
$ nnabla_cli convert mnistcnn.onnx mnistcnn.nnp 2021-07-27 13:40:17,220 [nnabla][INFO]: Initializing CPU extension... NNabla command line interface (Version:1.12.0, Build:200929110548) Traceback (most recent call last): File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/cli/cli.py", line 141, in cli_main return_value = args.func(args) File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/cli/convert.py", line 54, in convert_command nnabla.utils.converter.convert_files(args, args.files, output) File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/commands.py", line 260, in convert_files nnp = _import_file(args, ifiles) File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/commands.py", line 51, in _import_file return OnnxImporter(*ifiles).execute() File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 3002, in execute return self.onnx_model_to_nnp_protobuf() File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 2986, in onnx_model_to_nnp_protobuf self.onnx_graph_to_nnp_protobuf(pb) File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 2867, in onnx_graph_to_nnp_protobuf fl = self.convert_to_functions(n) File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 2857, in convert_to_functions ft(func_list, n) File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 1315, in Gemm bias_shape = [1] * (len(shape) - len(bias_shape)) + bias_shape TypeError: can only concatenate list (not "google.protobuf.pyext._message.RepeatedScalarContainer") to list
-
@nicibume I have another idea. Please use the latest version of NNabla, but update the NNabla C Runtime Library used by Spresense.
To do this:
- Update NNabla C Runtime Library:
$ cd externals/nnablart/nnabla-c-runtime $ git fetch $ git rebase origin/master
- Update nnablart Makefile:
diff --git a/externals/nnablart/Makefile b/externals/nnablart/Makefile index cb136a93..1182ad5b 100644 --- a/externals/nnablart/Makefile +++ b/externals/nnablart/Makefile @@ -56,7 +56,7 @@ CFLAGS += -std=c99 CFLAGS := $(patsubst -O%,-O3,$(CFLAGS)) -CSRCS = accessor.c list.c shape.c +CSRCS = accessor.c fixedpoint.c list.c shape.c ROOTDEPPATH = --dep-path $(SRCDIR)/functions/utilities VPATH = $(SRCDIR)/functions/utilities @@ -73,7 +73,7 @@ CSRCS += binary_tanh.c binary_sigmoid.c binary_connect_affine.c binary_weight_ ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/quantization VPATH += $(SRCDIR)/functions/implements/quantization -CSRCS += add_scalar.c arithmetic.c arithmetic_generic.c div2.c mul2.c mul_scalar.c +CSRCS += add_scalar.c arithmetic.c arithmetic_fixed.c arithmetic_generic.c div2.c mul2.c mul_scalar.c CSRCS += pow2.c pow_scalar.c r_div_scalar.c r_pow_scalar.c r_sub_scalar.c CSRCS += sub2.c add2.c ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/arithmetic @@ -92,11 +92,11 @@ CSRCS += pooling.c max_pooling.c sum_pooling.c average_pooling.c unpooling.c d ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/neural_network VPATH += $(SRCDIR)/functions/implements/neural_network -CSRCS += affine.c affine_generic.c +CSRCS += affine.c affine_fixed8.c affine_fixed16.c affine_generic.c ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/neural_network/affine VPATH += $(SRCDIR)/functions/implements/neural_network/affine -CSRCS += convolution.c convolution_generic.c convolution_float.c convolution_common.c +CSRCS += convolution.c convolution_generic.c convolution_float.c convolution_int8.c convolution_int16.c convolution_common.c CSRCS += binary_connect_convolution.c binary_weight_convolution.c depthwise_convolution.c ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/neural_network/convolution VPATH += $(SRCDIR)/functions/implements/neural_network/convolution diff --git a/externals/nnablart/nnabla-c-runtime b/externals/nnablart/nnabla-c-runtime index c58c11dd..a5fb10ea 160000 --- a/externals/nnablart/nnabla-c-runtime +++ b/externals/nnablart/nnabla-c-runtime @@ -1 +1 @@ -Subproject commit c58c11ddd182e8ed04eb931bbf2a82f190874c9d +Subproject commit a5fb10ea5f2e4915fc68823500d4308ab98eb473
Please let me know if it worked for you.
Best Regards,
Kamil Tomaszewski -
@kamiltomaszewski unfortunatelly it can't compile anymore. I'll dump the error:
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `exec_add_scalar_fixed16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add_scalar.c:80: undefined reference to `calc_scalar_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `saturate32_to_16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:93: undefined reference to `calc_add_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `exec_add_scalar_fixed8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add_scalar.c:91: undefined reference to `calc_scalar_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `saturate16_to_8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:71: undefined reference to `calc_add_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(div2.o): In function `exec_div2_fixed8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:88: undefined reference to `calc_arithmetic_fixed8_largebuff' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:91: undefined reference to `calc_div_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(div2.o): In function `exec_div2_fixed16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:99: undefined reference to `calc_arithmetic_fixed16_largebuff' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:102: undefined reference to `calc_div_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul2.o): In function `exec_mul2_fixed8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:96: undefined reference to `calc_arithmetic_fixed8_largebuff' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:99: undefined reference to `calc_mul_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul2.o): In function `exec_mul2_fixed16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:107: undefined reference to `calc_arithmetic_fixed16_largebuff' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:110: undefined reference to `calc_mul_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul_scalar.o): In function `exec_mul_scalar_fixed8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:85: undefined reference to `calc_scalar_fixed8_largebuff' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:87: undefined reference to `calc_mul_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul_scalar.o): In function `exec_mul_scalar_fixed16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:94: undefined reference to `calc_scalar_fixed16_largebuff' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:96: undefined reference to `calc_mul_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `exec_sub2_fixed16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:86: undefined reference to `equalize_bits_fixed16' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:87: undefined reference to `calc_arithmetic_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `saturate32_to_16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:93: undefined reference to `calc_sub_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `exec_sub2_fixed8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:96: undefined reference to `equalize_bits_fixed8' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:97: undefined reference to `calc_arithmetic_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `saturate16_to_8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:71: undefined reference to `calc_sub_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `exec_add2_fixed16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:87: undefined reference to `equalize_bits_fixed16' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:88: undefined reference to `calc_arithmetic_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `saturate32_to_16': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:93: undefined reference to `calc_add_fixed16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `exec_add2_fixed8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:97: undefined reference to `equalize_bits_fixed8' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:98: undefined reference to `calc_arithmetic_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `saturate16_to_8': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:71: undefined reference to `calc_add_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(affine.o): In function `allocate_affine_local_context': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/affine/affine.c:111: undefined reference to `exec_affine_fixed16' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/affine/affine.c:111: undefined reference to `exec_affine_fixed8' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(convolution.o): In function `allocate_convolution_local_context': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/convolution.c:40: undefined reference to `exec_convolution_int8' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/convolution.c:40: undefined reference to `exec_convolution_int16' /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(depthwise_convolution.o): In function `allocate_depthwise_convolution_local_context': /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:160: undefined reference to `exec_convolution_int16' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:160: undefined reference to `exec_convolution_int8' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:53: undefined reference to `exec_convolution_int8' /home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:53: undefined reference to `exec_convolution_int16' make[2]: *** [Makefile:185: nuttx] Error 1 make[2]: Leaving directory '/home/nicolas/spresense/nuttx/arch/arm/src' make[1]: *** [tools/Makefile.unix:464: pass2] Error 2 make[1]: Leaving directory '/home/nicolas/spresense/nuttx' make: *** [Makefile:99: all] Error 2
However, I want to inform you that the urgency of this problem is not too high (at least on my side) ^^
As my team and I are pursuing a different workaround altogether.Best,
Nic -
@niciBume You get these errors because you didn't make the changes I showed in step 2 to the
spresense/externals/nnablart/Makefile
.This adds missing files to the Makefile that were added in the new version of the NNabla C Runtime library.
You can use the code from step 2 as
.patch
and apply it to your Spresense SDK repository.Best Regards,
Kamil Tomaszewski -
@kamiltomaszewski oh, my bad. I misinterpreted step 2. Applied the patch and it compiles once again. During inference however, the error is again the same as in:
@nicibume said in How to convert from pytorch to nnb for spresense?:
Hi @kamiltomaszewski thanks for the reply!
The output of the dnnrt_lenet example is the following:
nsh> dnnrt_lenet --s /mnt/sd0/sony_nabla.nnb Load nnb file: /mnt/sd0/sony_nabla.nnb Load pgm image: /mnt/sd0/0.pgm Image Normalization (1.0/255.0): skipped start dnn_runtime_forward() up_hardfault: PANIC!!! Hard fault: 40000000 up_assert: Assertion failed at file:armv7-m/up_hardfault.c line: 148 task: sony_uav up_registerdump: R0: 0d0f6e54 0d09969c 0d099918 00000000 00000015 0d04b770 0d03d6b0 0d04b6e0 up_registerdump: R8: 00000003 00000000 00000000 00000000 00000009 0d04b6b0 0d024ae5 00000000 up_registerdump: xPSR: 00000000 BASEPRI: 000000e0 CONTROL: 00000000 up_registerdump: EXC_RETURN: ffffffe9 up_dumpstate: sp: 0d03aba8 up_dumpstate: IRQ stack: up_dumpstate: base: 0d03ac00 up_dumpstate: size: 00000800 up_dumpstate: used: 00000148 up_stackdump: 0d03aba0: 000007c4 0d00257b 000000e0 00000000 00000000 00000009 0d04b6b0 0d024ae5 up_stackdump: 0d03abc0: 00000000 0d03abd0 0d001a9f 00000003 00000000 0d001aa7 0d001a85 0d00354f up_stackdump: 0d03abe0: 000000e0 0d001a79 000000e0 0d04b5dc 0d04b770 0d03d6b0 0d04b6e0 0d000333 up_dumpstate: sp: 0d04b6b0 up_dumpstate: User stack: up_dumpstate: base: 0d04b720 up_dumpstate: size: 000007c4 up_dumpstate: used: 00000628 up_stackdump: 0d04b6a0: 00000000 80000000 80000010 0d024ae5 0d0f69a0 0d0f6900 0d04b770 0d020b7d up_stackdump: 0d04b6c0: 00015419 00000000 0d04ba00 0d03d6b0 00000001 0d020899 00000000 00000001 up_stackdump: 0d04b6e0: 0d03d6b0 0d04b770 00000249 00015419 00000000 00000000 00000000 00000101 up_stackdump: 0d04b700: 00000000 00000000 00000000 00000000 00000000 0d004377 00000000 00000000 up_taskdump: Idle Task: PID=0 Stack Used=0 of 0 up_taskdump: hpwork: PID=1 Stack Used=576 of 2028 up_taskdump: lpwork: PID=2 Stack Used=352 of 2028 up_taskdump: lpwork: PID=3 Stack Used=352 of 2028 up_taskdump: lpwork: PID=4 Stack Used=352 of 2028 up_taskdump: cxd56_pm_task: PID=6 Stack Used=352 of 996 up_taskdump: init: PID=7 Stack Used=1008 of 8172 up_taskdump: sony_uav: PID=16 Stack Used=1576 of 1988
This seems to be a nuttX error message, if I am not mistaking?
It actually passes
dnn_runtime_initialize
but fails atdnn_runtime_forward()
, where I trace it back to https://github.com/sonydevworld/spresense/blob/87f81d9c35c7d05d6a75e9177b45d6cbe8d8c063/sdk/modules/dnnrt/src/runtime/runtime_nnabla.c#L186Regarding the conversion, I tied your suggestion, yet with the same result. Would you know the build commands for the NNC?
Best,
Nic
-
@nicibume Could you please modify the
spresense/externals/nnablart/nnabla-c-runtime/src/runtime/runtime.c
file as shown below?diff --git a/src/runtime/runtime.c b/src/runtime/runtime.c index 6236510..2b45247 100644 --- a/src/runtime/runtime.c +++ b/src/runtime/runtime.c @@ -375,6 +375,9 @@ rt_return_value_t rt_forward(rt_context_pointer context) { rt_context_t *c = context; for (i = 0; i < c->num_of_functions; i++) { + printf("c->functions[i].func.exec_func %p\n", c->functions[i].func.exec_func); + printf("c->functions[i].info->type %d\n", c->functions[i].info->type); + usleep(1000); ret = c->functions[i].func.exec_func(&(c->functions[i].func)); if (ret != RT_FUNCTION_ERROR_NOERROR) { switch (ret) {
This adds 2 extra logs. Please send me an output from your serial console after making the change.
Best Regards,
Kamil Tomaszewski -
@kamiltomaszewski sorry for the late response, I must have missed the notification. I inserted your prints:
nsh> sony_uav /mnt/sd0/sony_nabla.nnb Load nnb file: /mnt/sd0/sony_nabla.nnb Load pgm image: /mnt/sd0/0.pgm Image Normalization (1.0/255.0): enabled INIT SUCCSESS!!! start dnn_runtime_forward() c->functions[i].func.exec_func 0xd02ff21 c->functions[i].info->type 126 c->functions[i].func.exec_func 0xd02ff21 c->functions[i].info->type 126 c->functions[i].func.exec_func 0xd0256cd c->functions[i].info->type 265 up_hardfault: PANIC!!! Hard fault: 40000000 up_assert: Assertion failed at file:armv7-m/up_hardfault.c line: 148 task: sony_uav up_registerdump: R0: 0d103114 6c46694f 0d10aa14 0d000305 1e8005f3 00000004 0d103114 0d045674 up_registerdump: R8: 0d045650 00000000 00000000 00000000 00008000 0d057b90 0d0256fd 0d025718 up_registerdump: xPSR: 21000000 BASEPRI: 000000e0 CONTROL: 00000000 up_registerdump: EXC_RETURN: ffffffe9 up_dumpstate: sp: 0d046ba8 up_dumpstate: IRQ stack: up_dumpstate: base: 0d046c00 up_dumpstate: size: 00000800 up_dumpstate: used: 00000148 up_stackdump: 0d046ba0: 000007cc 0d0027db 000000e0 00000000 00000000 00008000 0d057b90 0d0256fd up_stackdump: 0d046bc0: 0d025718 0d046bd0 0d001c67 00000003 00000000 0d001c6f 0d001c4d 0d0037af up_stackdump: 0d046be0: 000000e0 0d001c41 000000e0 0d057abc 00000004 0d103114 0d045674 0d000333 up_dumpstate: sp: 0d057b90 up_dumpstate: User stack: up_dumpstate: base: 0d057cb8 up_dumpstate: size: 000007cc up_dumpstate: used: 00000620 up_stackdump: 0d057b80: 00000000 00000000 00000000 0d102a74 00000000 00000000 00000000 00000000 up_stackdump: 0d057ba0: 00000000 00000000 00000014 0d050430 00000000 0d0057e3 0d050430 0a050730 up_stackdump: 0d057bc0: 0d050970 00000000 35323635 0d057c38 000000e0 0d057c38 00000000 0d04a840 up_stackdump: 0d057be0: 00000e18 00000000 00000000 0d00591f 00000000 0000000f 0d057c38 0d057c38 up_stackdump: 0d057c00: 00000002 00000000 0d045674 0d045650 00000000 0d0059c9 00000000 00000000 up_stackdump: 0d057c20: 0d057c38 0d0b54d0 00000002 00000040 0d045674 0d045650 00000000 00000000 up_stackdump: 0d057c40: 00000000 0d02af61 0d103100 0d0b54d0 0d0496d8 0d057c80 00000002 0d024cad up_stackdump: 0d057c60: 00038553 00000000 0d0496d8 0d0b5760 00000000 0d024965 00000000 00000001 up_stackdump: 0d057c80: 0d0496d8 0d0b54d0 00000025 00038553 00000000 00000000 00000101 00000000 up_stackdump: 0d057ca0: 00000000 00000000 00000000 0d0045db 00000000 00000000 deadbeef 0d057cc8 up_taskdump: Idle Task: PID=0 Stack Used=0 of 0 up_taskdump: hpwork: PID=1 Stack Used=576 of 2028 up_taskdump: lpwork: PID=2 Stack Used=352 of 2028 up_taskdump: lpwork: PID=3 Stack Used=352 of 2028 up_taskdump: lpwork: PID=4 Stack Used=352 of 2028 up_taskdump: cxd56_pm_task: PID=6 Stack Used=352 of 996 up_taskdump: init: PID=7 Stack Used=1056 of 8172 up_taskdump: sony_uav: PID=9 Stack Used=1568 of 1996
To compare, this is the output of a NNC exported .nnb:
nsh> sony_uav Load nnb file: /mnt/sd0/lenet-5.nnb Load pgm image: /mnt/sd0/0.pgm Image Normalization (1.0/255.0): enabled INIT SUCCSESS!!! start dnn_runtime_forward() c->functions[i].func.exec_func 0xd0256cd c->functions[i].info->type 1 c->functions[i].func.exec_func 0xd02b285 c->functions[i].info->type 14 c->functions[i].func.exec_func 0xd031aed c->functions[i].info->type 5 c->functions[i].func.exec_func 0xd0256cd c->functions[i].info->type 1 c->functions[i].func.exec_func 0xd031aed c->functions[i].info->type 5 c->functions[i].func.exec_func 0xd02b575 c->functions[i].info->type 13 c->functions[i].func.exec_func 0xd025141 c->functions[i].info->type 0 c->functions[i].func.exec_func 0xd02b285 c->functions[i].info->type 14 c->functions[i].func.exec_func 0xd025141 c->functions[i].info->type 0 c->functions[i].func.exec_func 0xd02b679 c->functions[i].info->type 16 Done with forward! output[0]=0.999999 output[1]=0.000000 output[2]=0.000000 output[3]=0.000000 output[4]=0.000000 output[5]=0.000000 output[6]=0.000000 output[7]=0.000000 output[8]=0.000000 output[9]=0.000001 inference time=0.272
Best,
Nic -
@nicibume Could you please share this model (
sony_nabla.nnb
) with me?My e-mail address: kamil.tomaszewski@sony.com
Best Regards,
Kamil Tomaszewski