Sony's Developer World forum
    • Home
    • Forum guidelines

    How to convert from pytorch to nnb for spresense?

    Spresense
    2
    14
    855
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      niciBume last edited by

      Hi!

      I want to train a NN in PyTorch and deploy the model on the spresense using the DNNRT library.

      According to the NNabla docs there is a way to convert NNs from ONNX to NNB.

      9be1fc80-2597-4c5a-9d11-8aead5eb8075-image.png

      I tested this conversion and tried to deploy it on the spresense in the same fashion as in the DNNRT example. However it failed to initialize the runtime ("Runtime initialization failure") with errorcode -1.

      To test if the conversion works properly, I exported a working model from the NNC as ONNX and tried the conversion with the following commands:

      nnabla_cli convert -b 1 sony.onnx sony_nabla.nnp
nnabla_cli convert -b 1 sony_nabla.nnp sony_nabla.nnb

      However this ended with the same runtime initialisation error as before. From here I conclude that I am missing the correct conversion arguments for nnabla_cli, could you help me out with this?

      Thanks a lot! 😊

      K 1 Reply Last reply Reply Quote
      • K
        KamilTomaszewski DeveloperWorld @niciBume last edited by

        Hi @niciBume

        Could you please try to find out where exactly dnn_runtime_initialize returns error?
        https://github.com/sonydevworld/spresense/blob/87f81d9c35c7d05d6a75e9177b45d6cbe8d8c063/sdk/modules/dnnrt/src/runtime/runtime_nnabla.c#L66

        Without it, it's hard to say what exactly doesn't work because this error only says that something didn't work in the NNabla C-runtime library.

        As for the conversion, I think you should use these commands. This is how it is described in the documentation:

        $ nnabla_cli convert sony.onnx sony_nabla.nnp
$ nnabla_cli convert -b 1 sony_nabla.nnp sony_nabla.nnb

        Best Regards,
        Kamil Tomaszewski

        N 1 Reply Last reply Reply Quote
        • N
          niciBume @KamilTomaszewski last edited by

          Hi @kamiltomaszewski thanks for the reply!

          The output of the dnnrt_lenet example is the following:

          nsh> dnnrt_lenet --s /mnt/sd0/sony_nabla.nnb
Load nnb file: /mnt/sd0/sony_nabla.nnb
Load pgm image: /mnt/sd0/0.pgm
Image Normalization (1.0/255.0): skipped
start dnn_runtime_forward()
up_hardfault: PANIC!!! Hard fault: 40000000
up_assert: Assertion failed at file:armv7-m/up_hardfault.c line: 148 task: sony_uav
up_registerdump: R0: 0d0f6e54 0d09969c 0d099918 00000000 00000015 0d04b770 0d03d6b0 0d04b6e0
up_registerdump: R8: 00000003 00000000 00000000 00000000 00000009 0d04b6b0 0d024ae5 00000000
up_registerdump: xPSR: 00000000 BASEPRI: 000000e0 CONTROL: 00000000
up_registerdump: EXC_RETURN: ffffffe9
up_dumpstate: sp:     0d03aba8
up_dumpstate: IRQ stack:
up_dumpstate:   base: 0d03ac00
up_dumpstate:   size: 00000800
up_dumpstate:   used: 00000148
up_stackdump: 0d03aba0: 000007c4 0d00257b 000000e0 00000000 00000000 00000009 0d04b6b0 0d024ae5
up_stackdump: 0d03abc0: 00000000 0d03abd0 0d001a9f 00000003 00000000 0d001aa7 0d001a85 0d00354f
up_stackdump: 0d03abe0: 000000e0 0d001a79 000000e0 0d04b5dc 0d04b770 0d03d6b0 0d04b6e0 0d000333
up_dumpstate: sp:     0d04b6b0
up_dumpstate: User stack:
up_dumpstate:   base: 0d04b720
up_dumpstate:   size: 000007c4
up_dumpstate:   used: 00000628
up_stackdump: 0d04b6a0: 00000000 80000000 80000010 0d024ae5 0d0f69a0 0d0f6900 0d04b770 0d020b7d
up_stackdump: 0d04b6c0: 00015419 00000000 0d04ba00 0d03d6b0 00000001 0d020899 00000000 00000001
up_stackdump: 0d04b6e0: 0d03d6b0 0d04b770 00000249 00015419 00000000 00000000 00000000 00000101
up_stackdump: 0d04b700: 00000000 00000000 00000000 00000000 00000000 0d004377 00000000 00000000
up_taskdump: Idle Task: PID=0 Stack Used=0 of 0
up_taskdump: hpwork: PID=1 Stack Used=576 of 2028
up_taskdump: lpwork: PID=2 Stack Used=352 of 2028
up_taskdump: lpwork: PID=3 Stack Used=352 of 2028
up_taskdump: lpwork: PID=4 Stack Used=352 of 2028
up_taskdump: cxd56_pm_task: PID=6 Stack Used=352 of 996
up_taskdump: init: PID=7 Stack Used=1008 of 8172
up_taskdump: sony_uav: PID=16 Stack Used=1576 of 1988

          This seems to be a nuttX error message, if I am not mistaking?

          It actually passes dnn_runtime_initialize but fails at dnn_runtime_forward(), where I trace it back to https://github.com/sonydevworld/spresense/blob/87f81d9c35c7d05d6a75e9177b45d6cbe8d8c063/sdk/modules/dnnrt/src/runtime/runtime_nnabla.c#L186

          Regarding the conversion, I tied your suggestion, yet with the same result. Would you know the build commands for the NNC?

          Best,

          Nic

          K 1 Reply Last reply Reply Quote
          • K
            KamilTomaszewski DeveloperWorld @niciBume last edited by

            @nicibume

            Could you please try the converter from Nnabla version 1.12.0?

            Best Regards,
            Kamil Tomaszewski

            N 2 Replies Last reply Reply Quote
            • N
              niciBume @KamilTomaszewski last edited by

              @kamiltomaszewski

              I downgraded Nnabla to 1.12.0 and it worked! Thanks a lot for the help!

              Cheers,
              Nic

              1 Reply Last reply Reply Quote
              • N
                niciBume @KamilTomaszewski last edited by

                @kamiltomaszewski

                Hmmm... while the example (exported it directly from NNC as .onnx and .nnb to ensure that the model works) .onnx -> .nnp -> .nnb now works on the Spresense. However, with Nnabla 1.12.0 my actual PyTorch -> .onnx model can not be converted to .nnp anymore. This is a step that actually worked with Nnabla 1.20.0. I traced this issue back to Nnabla 1.14.0, where PyTorch -> .onnx works again, but the .onnx -> .nnp -> .nnb conversion yields the same error on the MCU as previously stated.

                Is there a workaround for this?

                Best,
                Nic

                N 1 Reply Last reply Reply Quote
                • N
                  niciBume @niciBume last edited by niciBume

                  Forgot to add error print (and can't edit post anymore):

                  $ nnabla_cli convert mnistcnn.onnx mnistcnn.nnp
2021-07-27 13:40:17,220 [nnabla][INFO]: Initializing CPU extension...
NNabla command line interface (Version:1.12.0, Build:200929110548)
Traceback (most recent call last):
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/cli/cli.py", line 141, in cli_main
    return_value = args.func(args)
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/cli/convert.py", line 54, in convert_command
    nnabla.utils.converter.convert_files(args, args.files, output)
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/commands.py", line 260, in convert_files
    nnp = _import_file(args, ifiles)
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/commands.py", line 51, in _import_file
    return OnnxImporter(*ifiles).execute()
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 3002, in execute
    return self.onnx_model_to_nnp_protobuf()
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 2986, in onnx_model_to_nnp_protobuf
    self.onnx_graph_to_nnp_protobuf(pb)
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 2867, in onnx_graph_to_nnp_protobuf
    fl = self.convert_to_functions(n)
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 2857, in convert_to_functions
    ft(func_list, n)
  File "/home/niclas/anaconda3/envs/torch_nnp_test/lib/python3.6/site-packages/nnabla/utils/converter/onnx/importer.py", line 1315, in Gemm
    bias_shape = [1] * (len(shape) - len(bias_shape)) + bias_shape
TypeError: can only concatenate list (not "google.protobuf.pyext._message.RepeatedScalarContainer") to list
                  K 1 Reply Last reply Reply Quote
                  • K
                    KamilTomaszewski DeveloperWorld @niciBume last edited by

                    @nicibume I have another idea. Please use the latest version of NNabla, but update the NNabla C Runtime Library used by Spresense.

                    To do this:

                    1. Update NNabla C Runtime Library:
                    $ cd externals/nnablart/nnabla-c-runtime
$ git fetch
$ git rebase origin/master
                    1. Update nnablart Makefile:
                    diff --git a/externals/nnablart/Makefile b/externals/nnablart/Makefile
index cb136a93..1182ad5b 100644
--- a/externals/nnablart/Makefile
+++ b/externals/nnablart/Makefile
@@ -56,7 +56,7 @@ CFLAGS += -std=c99
 
 CFLAGS := $(patsubst -O%,-O3,$(CFLAGS))
 
-CSRCS    =   accessor.c list.c shape.c
+CSRCS    =   accessor.c fixedpoint.c list.c shape.c
 ROOTDEPPATH  = --dep-path $(SRCDIR)/functions/utilities
 VPATH    = $(SRCDIR)/functions/utilities
 
@@ -73,7 +73,7 @@ CSRCS   += binary_tanh.c binary_sigmoid.c binary_connect_affine.c binary_weight_
 ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/quantization
 VPATH   += $(SRCDIR)/functions/implements/quantization
 
-CSRCS   += add_scalar.c arithmetic.c arithmetic_generic.c div2.c mul2.c mul_scalar.c
+CSRCS   += add_scalar.c arithmetic.c arithmetic_fixed.c arithmetic_generic.c div2.c mul2.c mul_scalar.c
 CSRCS   += pow2.c pow_scalar.c r_div_scalar.c r_pow_scalar.c r_sub_scalar.c
 CSRCS   += sub2.c add2.c
 ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/arithmetic
@@ -92,11 +92,11 @@ CSRCS   += pooling.c max_pooling.c sum_pooling.c average_pooling.c unpooling.c d
 ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/neural_network
 VPATH   += $(SRCDIR)/functions/implements/neural_network
 
-CSRCS   += affine.c affine_generic.c
+CSRCS   += affine.c affine_fixed8.c affine_fixed16.c affine_generic.c
 ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/neural_network/affine
 VPATH   += $(SRCDIR)/functions/implements/neural_network/affine
 
-CSRCS   += convolution.c convolution_generic.c convolution_float.c convolution_common.c
+CSRCS   += convolution.c convolution_generic.c convolution_float.c convolution_int8.c convolution_int16.c convolution_common.c
 CSRCS   += binary_connect_convolution.c binary_weight_convolution.c depthwise_convolution.c
 ROOTDEPPATH += --dep-path $(SRCDIR)/functions/implements/neural_network/convolution
 VPATH   += $(SRCDIR)/functions/implements/neural_network/convolution
diff --git a/externals/nnablart/nnabla-c-runtime b/externals/nnablart/nnabla-c-runtime
index c58c11dd..a5fb10ea 160000
--- a/externals/nnablart/nnabla-c-runtime
+++ b/externals/nnablart/nnabla-c-runtime
@@ -1 +1 @@
-Subproject commit c58c11ddd182e8ed04eb931bbf2a82f190874c9d
+Subproject commit a5fb10ea5f2e4915fc68823500d4308ab98eb473

                    Please let me know if it worked for you.

                    Best Regards,
                    Kamil Tomaszewski

                    N 1 Reply Last reply Reply Quote
                    • N
                      niciBume @KamilTomaszewski last edited by

                      @kamiltomaszewski unfortunatelly it can't compile anymore. I'll dump the error:

                      /home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `exec_add_scalar_fixed16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add_scalar.c:80: undefined reference to `calc_scalar_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `saturate32_to_16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:93: undefined reference to `calc_add_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `exec_add_scalar_fixed8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add_scalar.c:91: undefined reference to `calc_scalar_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add_scalar.o): In function `saturate16_to_8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:71: undefined reference to `calc_add_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(div2.o): In function `exec_div2_fixed8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:88: undefined reference to `calc_arithmetic_fixed8_largebuff'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:91: undefined reference to `calc_div_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(div2.o): In function `exec_div2_fixed16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:99: undefined reference to `calc_arithmetic_fixed16_largebuff'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/div2.c:102: undefined reference to `calc_div_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul2.o): In function `exec_mul2_fixed8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:96: undefined reference to `calc_arithmetic_fixed8_largebuff'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:99: undefined reference to `calc_mul_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul2.o): In function `exec_mul2_fixed16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:107: undefined reference to `calc_arithmetic_fixed16_largebuff'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul2.c:110: undefined reference to `calc_mul_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul_scalar.o): In function `exec_mul_scalar_fixed8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:85: undefined reference to `calc_scalar_fixed8_largebuff'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:87: undefined reference to `calc_mul_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(mul_scalar.o): In function `exec_mul_scalar_fixed16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:94: undefined reference to `calc_scalar_fixed16_largebuff'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/mul_scalar.c:96: undefined reference to `calc_mul_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `exec_sub2_fixed16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:86: undefined reference to `equalize_bits_fixed16'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:87: undefined reference to `calc_arithmetic_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `saturate32_to_16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:93: undefined reference to `calc_sub_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `exec_sub2_fixed8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:96: undefined reference to `equalize_bits_fixed8'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/sub2.c:97: undefined reference to `calc_arithmetic_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(sub2.o): In function `saturate16_to_8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:71: undefined reference to `calc_sub_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `exec_add2_fixed16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:87: undefined reference to `equalize_bits_fixed16'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:88: undefined reference to `calc_arithmetic_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `saturate32_to_16':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:93: undefined reference to `calc_add_fixed16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `exec_add2_fixed8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:97: undefined reference to `equalize_bits_fixed8'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/add2.c:98: undefined reference to `calc_arithmetic_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(add2.o): In function `saturate16_to_8':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/arithmetic/../../utilities/fixedpoint.h:71: undefined reference to `calc_add_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(affine.o): In function `allocate_affine_local_context':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/affine/affine.c:111: undefined reference to `exec_affine_fixed16'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/affine/affine.c:111: undefined reference to `exec_affine_fixed8'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(convolution.o): In function `allocate_convolution_local_context':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/convolution.c:40: undefined reference to `exec_convolution_int8'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/convolution.c:40: undefined reference to `exec_convolution_int16'
/home/nicolas/spresense/nuttx/../externals/nnablart/libnnablart.a(depthwise_convolution.o): In function `allocate_depthwise_convolution_local_context':
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:160: undefined reference to `exec_convolution_int16'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:160: undefined reference to `exec_convolution_int8'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:53: undefined reference to `exec_convolution_int8'
/home/nicolas/spresense/externals/nnablart/nnabla-c-runtime/src/functions/implements/neural_network/convolution/depthwise_convolution.c:53: undefined reference to `exec_convolution_int16'
make[2]: *** [Makefile:185: nuttx] Error 1
make[2]: Leaving directory '/home/nicolas/spresense/nuttx/arch/arm/src'
make[1]: *** [tools/Makefile.unix:464: pass2] Error 2
make[1]: Leaving directory '/home/nicolas/spresense/nuttx'
make: *** [Makefile:99: all] Error 2

                      However, I want to inform you that the urgency of this problem is not too high (at least on my side) ^^
                      As my team and I are pursuing a different workaround altogether.

                      Best,
                      Nic

                      K 1 Reply Last reply Reply Quote
                      • K
                        KamilTomaszewski DeveloperWorld @niciBume last edited by

                        @niciBume You get these errors because you didn't make the changes I showed in step 2 to the spresense/externals/nnablart/Makefile.

                        This adds missing files to the Makefile that were added in the new version of the NNabla C Runtime library.

                        You can use the code from step 2 as .patch and apply it to your Spresense SDK repository.

                        Best Regards,
                        Kamil Tomaszewski

                        N 1 Reply Last reply Reply Quote
                        • N
                          niciBume @KamilTomaszewski last edited by

                          @kamiltomaszewski oh, my bad. I misinterpreted step 2. Applied the patch and it compiles once again. During inference however, the error is again the same as in:

                          @nicibume said in How to convert from pytorch to nnb for spresense?:

                          Hi @kamiltomaszewski thanks for the reply!

                          The output of the dnnrt_lenet example is the following:

                          nsh> dnnrt_lenet --s /mnt/sd0/sony_nabla.nnb
Load nnb file: /mnt/sd0/sony_nabla.nnb
Load pgm image: /mnt/sd0/0.pgm
Image Normalization (1.0/255.0): skipped
start dnn_runtime_forward()
up_hardfault: PANIC!!! Hard fault: 40000000
up_assert: Assertion failed at file:armv7-m/up_hardfault.c line: 148 task: sony_uav
up_registerdump: R0: 0d0f6e54 0d09969c 0d099918 00000000 00000015 0d04b770 0d03d6b0 0d04b6e0
up_registerdump: R8: 00000003 00000000 00000000 00000000 00000009 0d04b6b0 0d024ae5 00000000
up_registerdump: xPSR: 00000000 BASEPRI: 000000e0 CONTROL: 00000000
up_registerdump: EXC_RETURN: ffffffe9
up_dumpstate: sp:     0d03aba8
up_dumpstate: IRQ stack:
up_dumpstate:   base: 0d03ac00
up_dumpstate:   size: 00000800
up_dumpstate:   used: 00000148
up_stackdump: 0d03aba0: 000007c4 0d00257b 000000e0 00000000 00000000 00000009 0d04b6b0 0d024ae5
up_stackdump: 0d03abc0: 00000000 0d03abd0 0d001a9f 00000003 00000000 0d001aa7 0d001a85 0d00354f
up_stackdump: 0d03abe0: 000000e0 0d001a79 000000e0 0d04b5dc 0d04b770 0d03d6b0 0d04b6e0 0d000333
up_dumpstate: sp:     0d04b6b0
up_dumpstate: User stack:
up_dumpstate:   base: 0d04b720
up_dumpstate:   size: 000007c4
up_dumpstate:   used: 00000628
up_stackdump: 0d04b6a0: 00000000 80000000 80000010 0d024ae5 0d0f69a0 0d0f6900 0d04b770 0d020b7d
up_stackdump: 0d04b6c0: 00015419 00000000 0d04ba00 0d03d6b0 00000001 0d020899 00000000 00000001
up_stackdump: 0d04b6e0: 0d03d6b0 0d04b770 00000249 00015419 00000000 00000000 00000000 00000101
up_stackdump: 0d04b700: 00000000 00000000 00000000 00000000 00000000 0d004377 00000000 00000000
up_taskdump: Idle Task: PID=0 Stack Used=0 of 0
up_taskdump: hpwork: PID=1 Stack Used=576 of 2028
up_taskdump: lpwork: PID=2 Stack Used=352 of 2028
up_taskdump: lpwork: PID=3 Stack Used=352 of 2028
up_taskdump: lpwork: PID=4 Stack Used=352 of 2028
up_taskdump: cxd56_pm_task: PID=6 Stack Used=352 of 996
up_taskdump: init: PID=7 Stack Used=1008 of 8172
up_taskdump: sony_uav: PID=16 Stack Used=1576 of 1988

                          This seems to be a nuttX error message, if I am not mistaking?

                          It actually passes dnn_runtime_initialize but fails at dnn_runtime_forward(), where I trace it back to https://github.com/sonydevworld/spresense/blob/87f81d9c35c7d05d6a75e9177b45d6cbe8d8c063/sdk/modules/dnnrt/src/runtime/runtime_nnabla.c#L186

                          Regarding the conversion, I tied your suggestion, yet with the same result. Would you know the build commands for the NNC?

                          Best,

                          Nic

                          K 1 Reply Last reply Reply Quote
                          • K
                            KamilTomaszewski DeveloperWorld @niciBume last edited by

                            @nicibume Could you please modify the spresense/externals/nnablart/nnabla-c-runtime/src/runtime/runtime.c file as shown below?

                            diff --git a/src/runtime/runtime.c b/src/runtime/runtime.c
index 6236510..2b45247 100644
--- a/src/runtime/runtime.c
+++ b/src/runtime/runtime.c
@@ -375,6 +375,9 @@ rt_return_value_t rt_forward(rt_context_pointer context) {
   rt_context_t *c = context;
 
   for (i = 0; i < c->num_of_functions; i++) {
+    printf("c->functions[i].func.exec_func %p\n", c->functions[i].func.exec_func);
+    printf("c->functions[i].info->type %d\n", c->functions[i].info->type);
+    usleep(1000);
     ret = c->functions[i].func.exec_func(&(c->functions[i].func));
     if (ret != RT_FUNCTION_ERROR_NOERROR) {
       switch (ret) {

                            This adds 2 extra logs. Please send me an output from your serial console after making the change.

                            Best Regards,
                            Kamil Tomaszewski

                            N 1 Reply Last reply Reply Quote
                            • N
                              niciBume @KamilTomaszewski last edited by

                              @kamiltomaszewski sorry for the late response, I must have missed the notification. I inserted your prints:

                              nsh> sony_uav /mnt/sd0/sony_nabla.nnb
Load nnb file: /mnt/sd0/sony_nabla.nnb
Load pgm image: /mnt/sd0/0.pgm
Image Normalization (1.0/255.0): enabled
INIT SUCCSESS!!!
start dnn_runtime_forward()
c->functions[i].func.exec_func 0xd02ff21
c->functions[i].info->type 126
c->functions[i].func.exec_func 0xd02ff21
c->functions[i].info->type 126
c->functions[i].func.exec_func 0xd0256cd
c->functions[i].info->type 265
up_hardfault: PANIC!!! Hard fault: 40000000
up_assert: Assertion failed at file:armv7-m/up_hardfault.c line: 148 task: sony_uav
up_registerdump: R0: 0d103114 6c46694f 0d10aa14 0d000305 1e8005f3 00000004 0d103114 0d045674
up_registerdump: R8: 0d045650 00000000 00000000 00000000 00008000 0d057b90 0d0256fd 0d025718
up_registerdump: xPSR: 21000000 BASEPRI: 000000e0 CONTROL: 00000000
up_registerdump: EXC_RETURN: ffffffe9
up_dumpstate: sp:     0d046ba8
up_dumpstate: IRQ stack:
up_dumpstate:   base: 0d046c00
up_dumpstate:   size: 00000800
up_dumpstate:   used: 00000148
up_stackdump: 0d046ba0: 000007cc 0d0027db 000000e0 00000000 00000000 00008000 0d057b90 0d0256fd
up_stackdump: 0d046bc0: 0d025718 0d046bd0 0d001c67 00000003 00000000 0d001c6f 0d001c4d 0d0037af
up_stackdump: 0d046be0: 000000e0 0d001c41 000000e0 0d057abc 00000004 0d103114 0d045674 0d000333
up_dumpstate: sp:     0d057b90
up_dumpstate: User stack:
up_dumpstate:   base: 0d057cb8
up_dumpstate:   size: 000007cc
up_dumpstate:   used: 00000620
up_stackdump: 0d057b80: 00000000 00000000 00000000 0d102a74 00000000 00000000 00000000 00000000
up_stackdump: 0d057ba0: 00000000 00000000 00000014 0d050430 00000000 0d0057e3 0d050430 0a050730
up_stackdump: 0d057bc0: 0d050970 00000000 35323635 0d057c38 000000e0 0d057c38 00000000 0d04a840
up_stackdump: 0d057be0: 00000e18 00000000 00000000 0d00591f 00000000 0000000f 0d057c38 0d057c38
up_stackdump: 0d057c00: 00000002 00000000 0d045674 0d045650 00000000 0d0059c9 00000000 00000000
up_stackdump: 0d057c20: 0d057c38 0d0b54d0 00000002 00000040 0d045674 0d045650 00000000 00000000
up_stackdump: 0d057c40: 00000000 0d02af61 0d103100 0d0b54d0 0d0496d8 0d057c80 00000002 0d024cad
up_stackdump: 0d057c60: 00038553 00000000 0d0496d8 0d0b5760 00000000 0d024965 00000000 00000001
up_stackdump: 0d057c80: 0d0496d8 0d0b54d0 00000025 00038553 00000000 00000000 00000101 00000000
up_stackdump: 0d057ca0: 00000000 00000000 00000000 0d0045db 00000000 00000000 deadbeef 0d057cc8
up_taskdump: Idle Task: PID=0 Stack Used=0 of 0
up_taskdump: hpwork: PID=1 Stack Used=576 of 2028
up_taskdump: lpwork: PID=2 Stack Used=352 of 2028
up_taskdump: lpwork: PID=3 Stack Used=352 of 2028
up_taskdump: lpwork: PID=4 Stack Used=352 of 2028
up_taskdump: cxd56_pm_task: PID=6 Stack Used=352 of 996
up_taskdump: init: PID=7 Stack Used=1056 of 8172
up_taskdump: sony_uav: PID=9 Stack Used=1568 of 1996

                              To compare, this is the output of a NNC exported .nnb:

                              nsh> sony_uav
Load nnb file: /mnt/sd0/lenet-5.nnb
Load pgm image: /mnt/sd0/0.pgm
Image Normalization (1.0/255.0): enabled
INIT SUCCSESS!!!
start dnn_runtime_forward()
c->functions[i].func.exec_func 0xd0256cd
c->functions[i].info->type 1
c->functions[i].func.exec_func 0xd02b285
c->functions[i].info->type 14
c->functions[i].func.exec_func 0xd031aed
c->functions[i].info->type 5
c->functions[i].func.exec_func 0xd0256cd
c->functions[i].info->type 1
c->functions[i].func.exec_func 0xd031aed
c->functions[i].info->type 5
c->functions[i].func.exec_func 0xd02b575
c->functions[i].info->type 13
c->functions[i].func.exec_func 0xd025141
c->functions[i].info->type 0
c->functions[i].func.exec_func 0xd02b285
c->functions[i].info->type 14
c->functions[i].func.exec_func 0xd025141
c->functions[i].info->type 0
c->functions[i].func.exec_func 0xd02b679
c->functions[i].info->type 16
Done with forward! 
output[0]=0.999999
output[1]=0.000000
output[2]=0.000000
output[3]=0.000000
output[4]=0.000000
output[5]=0.000000
output[6]=0.000000
output[7]=0.000000
output[8]=0.000000
output[9]=0.000001
inference time=0.272

                              Best,
                              Nic

                              K 1 Reply Last reply Reply Quote
                              • K
                                KamilTomaszewski DeveloperWorld @niciBume last edited by

                                @nicibume Could you please share this model (sony_nabla.nnb) with me?

                                My e-mail address: kamil.tomaszewski@sony.com

                                Best Regards,
                                Kamil Tomaszewski

                                1 Reply Last reply Reply Quote
                                • First post
                                  Last post
                                Developer World
                                Copyright © 2021 Sony Group Corporation. All rights reserved.
                                • Contact us
                                • Legal