Errors while parsing DBC #330

gianpyc · 2019-03-16T13:20:24Z

I get an error when parsing a DBC similar to:
BA_ "Period [ms]" BO_ 1234567890 3000;
and
BA_ "Signal Age [ms]" SG_ 1234567890 sometext 5000;

Is this a bug or an issue on the DBC file?
Thanks

The text was updated successfully, but these errors were encountered:

altendky · 2019-03-16T18:24:36Z

Are you able to share the problematic .dbc? Regardless, please share what version of canmatrix you are running and also the full traceback.

Cheers,
-kyle

gianpyc · 2019-03-16T18:45:39Z

Sorry I cannot share the dbc.
I installed canmatrix today using pip install canmatrix on mac os. I wasn't able to understand the version but pip download from:
https://files.pythonhosted.org/packages/a5/a1/b2bd32bb0e0898e8e80f5e3ef5f42c54c3dea95d9df05929d8b2a2e71b17/canmatrix-0.7-py2.py3-none-any.whl
plus other dependencies.

This is the command I run:
canconvert -v --frames=MY_ECU_NAME mydb.dbc target.dbc

The traceback is composed by a set of errors like:

error with line no: 810
BA_ "Period [ms]" BO_ 1234567890 3000;
...
error with line no: 998
BA_ "Signal Age [ms]" SG_ 1234567890 sometext 5000;

Plus other similar errors

and

INFO - convert - done
INFO - convert - Exporting target.dbc ... 
INFO - copy - Copying Frame MY_ECU_NAME
Traceback (most recent call last):
  File "/usr/local/bin/canconvert", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/canmatrix/convert.py", line 402, in main
    convert(infile, outfileName, **cmdlineOptions.__dict__)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/convert.py", line 61, in convert
    canmatrix.copy.copyFrame(frame, dbs[name], db)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/copy.py", line 134, in copyFrame
    copyBU(sourceBU, sourceDb, targetDb)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/copy.py", line 49, in copyBU
    deepcopy(attribute), deepcopy(sourceDb.buDefines[attribute].definition))
KeyError: u'Node Address'

Hope this helps.

Cheers

ebroecker · 2019-03-17T15:35:09Z

@gianpyc
Thank you very much for your bug report.

I could identify one problem...
Canmatrix has currently problems with braces [,] in attributes.

I have a local fix for this issue here.
could you please try:
pip install git+https://github.com/ebroecker/canmatrix@iss330#egg=canmatrix
and give feedback?

Thanks

gianpyc · 2019-03-17T17:19:39Z

with this command my repo is not updated and I get:

Requirement already satisfied: canmatrix from git+https://github.com/ebroecker/canmatrix@iss330#egg=canmatrix in /usr/local/lib/python2.7/site-packages (0.7)
Requirement already satisfied: bitstruct in /usr/local/lib/python2.7/site-packages (from canmatrix) (6.0.0)
Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python2.7/site-packages (from canmatrix) (19.1.0)
Requirement already satisfied: pathlib2 in /usr/local/lib/python2.7/site-packages (from canmatrix) (2.3.3)
Requirement already satisfied: future in /usr/local/lib/python2.7/site-packages (from canmatrix) (0.17.1)
Requirement already satisfied: scandir; python_version < "3.5" in /usr/local/lib/python2.7/site-packages (from pathlib2->canmatrix) (1.10.0)
Requirement already satisfied: six in /usr/local/lib/python2.7/site-packages (from pathlib2->canmatrix) (1.11.0)

So, I updated manually the file dpc.py but I wasn't able to update test_dbc.py. This last file seems different with the version that you modified.

Btw, when running the command again I got less errors but this traceback:

INFO - convert - Importing my.dbc ... 
INFO - convert - done

INFO - convert - Exporting target.dbc ... 
INFO - copy - Copying Frame My_ECU
Traceback (most recent call last):
  File "/usr/local/bin/canconvert", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/canmatrix/convert.py", line 402, in main
    convert(infile, outfileName, **cmdlineOptions.__dict__)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/convert.py", line 61, in convert
    canmatrix.copy.copyFrame(frame, dbs[name], db)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/copy.py", line 134, in copyFrame
    copyBU(sourceBU, sourceDb, targetDb)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/copy.py", line 49, in copyBU
    deepcopy(attribute), deepcopy(sourceDb.buDefines[attribute].definition))
KeyError: u'Node Address'

Thanks for your help

ebroecker · 2019-03-17T17:52:10Z

Thanks for trying the fix.

without digging in it seems a bit like there is no definition for Node Address.
Could you verify that in mydb.dbc there is a definition like:
BA_DEF BU_ "Node Address" STRING;

?

gianpyc · 2019-03-17T18:07:19Z

I have this definition:
BA_DEF_ BU_ "Node Address" INT 0 255;
and not that one suggested.

Trying this instead:
BA_DEF BU_ "Node Address" STRING;

Same error

Thanks

ebroecker · 2019-03-17T18:49:41Z

Thanks for looking into your dbc.

BA_DEF_ BU_ "Node Address" INT 0 255;

is surely right (the other one with STRING is wrong!)

I'm a bit clueless, the code which fails does nothing really complicated:

canmatrix/src/canmatrix/copy.py

Lines 46 to 49 in 5ccf848

    
           for attribute in bu.attributes: 
        
               if attribute not in targetDb.buDefines: 
        
                   targetDb.addBUDefines( 
        
                       deepcopy(attribute), deepcopy(sourceDb.buDefines[attribute].definition))

walk through all attributes of the given ECU and copy these attributes and their definitons to the new matrix. And it looks like the definition is not found. But you found it in your dbc...

I'll play a bit around, maybe I'll find some hint...

EDIT: I can reproduce now. please stay tuned for a fix.

fix for #330

ebroecker · 2019-03-17T19:25:31Z

could you please again fetch the dbc.py from the branch above and test?

gianpyc · 2019-03-17T20:01:02Z

Now I get this traceback:
Traceback (most recent call last):

  File "/usr/local/bin/canconvert", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/canmatrix/convert.py", line 402, in main
    convert(infile, outfileName, **cmdlineOptions.__dict__)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/convert.py", line 43, in convert
    dbs = canmatrix.formats.loadp(infile, **options)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/formats.py", line 64, in loadp
    return load(fileObject, importType, key, flatImport, **options)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/formats.py", line 76, in load
    dbs[key] = moduleInstance.load(fileObject, **options)
  File "/usr/local/lib/python2.7/site-packages/canmatrix/dbc.py", line 721, in load
    db.ecus.append(canmatrix.Ecu(ele))
AttributeError: 'CanMatrix' object has no attribute 'ecus'

Thanks

ebroecker · 2019-03-17T22:11:06Z

Hi @gianpyc

we did heavy refactoring the last days [ecus is the new boardUnits].
So please either try to get the whole branch iss330 or adapt only the change of the dbc.py ...

gianpyc · 2019-03-18T09:10:53Z

Thanks,
I updated canmatrix applying last changes and now I get this traceback:

Traceback (most recent call last):
  File "/usr/local/bin/canconvert", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/convert.py", line 411, in main
    convert(infile, out_file_name, **cmdlineOptions.__dict__)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/convert.py", line 42, in convert
    dbs = canmatrix.formats.loadp(infile, **options)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/formats/__init__.py", line 68, in loadp
    return load(fileObject, import_type, key, flat_import, **options)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/formats/__init__.py", line 81, in load
    dbs[key] = module_instance.load(file_object, **options)  # type: ignore
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/formats/dbc.py", line 794, in load
    if tempba.group(1).strip().startswith("BO_ "):
AttributeError: 'NoneType' object has no attribute 'group'

ebroecker · 2019-03-18T20:06:22Z

seems you have a lot of special characters in attribute names ... could you please try once more?

gianpyc · 2019-03-19T09:06:05Z

No I get this traceback :(

Traceback (most recent call last):
  File "/usr/local/bin/canconvert", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/convert.py", line 411, in main
    convert(infile, out_file_name, **cmdlineOptions.__dict__)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/convert.py", line 42, in convert
    dbs = canmatrix.formats.loadp(infile, **options)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/formats/__init__.py", line 68, in loadp
    return load(fileObject, import_type, key, flat_import, **options)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/formats/__init__.py", line 81, in load
    dbs[key] = module_instance.load(file_object, **options)  # type: ignore
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/formats/dbc.py", line 798, in load
    print('group3', temp.group(3))
IndexError: no such group

From a quick debug I saw that temp.group(2) and temp.group(3) are empty. Only temp.group(1) has a content.

Thanks for your support

ebroecker · 2019-03-19T09:14:52Z

Hi @gianpyc,
Thanks for testing.

Did you upgrade branch "iss330" or did you only apply
8057942 ?

I activated the global "try" "except" again in 33f0259, this should print out the failing input line in dbc.
With this line, I hope I can reproduce the issue...

EDIT: If you are debugging, just give me the content of decoded...

gianpyc · 2019-03-19T09:28:15Z

Before I did git pull but I saw that it didn't change a lot the dbc.py code so I manually made
8057942

Now, I replaced all the code that I have in dbc.py with last one you updated:
https://raw.githubusercontent.com/ebroecker/canmatrix/33f0259031bdfad555fd4d8220d2d85e2903a815/src/canmatrix/formats/dbc.py

And I get this traceback:

INFO - convert - Importing MYdbc.dbc ... 
INFO - convert - done

INFO - convert - Exporting target.dbc ... 
INFO - copy - Copying Frame MY_ECU
Traceback (most recent call last):
  File "/usr/local/bin/canconvert", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/convert.py", line 411, in main
    convert(infile, out_file_name, **cmdlineOptions.__dict__)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/convert.py", line 61, in convert
    canmatrix.copy.copy_frame(frame_to_copy.arbitration_id, dbs[name], db)
  File "/usr/local/lib/python2.7/dist-packages/canmatrix/copy.py", line 175, in copy_frame
    copy.deepcopy(attribute), copy.deepcopy(source_db.frame_defines[attribute].definition))
KeyError: u'Period [ms]'

Thanks

ebroecker · 2019-03-19T10:00:13Z

ok, thanks, I'm gonna work on this, looks quite similar to the initial bug...

ebroecker · 2019-03-19T23:21:08Z

ok, it's worth another try

gianpyc · 2019-03-20T09:03:02Z

Much better, now I reach the end of the operation if even this errors occurs during the parsing:

BA_ "Description" BU_ ECU "SOME TEXT";
BA_ "Accuracy" SG_ Number_ID TEXT "+/- 0.12 V";
BA_ "Accuracy" SG_  Number_ID Some_TEXT "+/- 0.9 bar";
BA_ "Accuracy" SG_ Number_ID Text "+/- 10.2 at 55.1%";

Plus other similar errors

In addition, I'd like to ask if with canmatrix it is possible to extract signal frame in its hex format, as it'd be sent by an ECU in the CAN payload, by setting manually the signal value or by having all combination for those signal values. It'd be very nice to have as feature.

Thanks a lot for your support

ebroecker · 2019-03-20T20:23:11Z

@gianpyc
thanks for your patience.
I did more repairing on your provided reports.

Yes - it is possible to decode can-data and to encode can-data.

have a look at Frame.decode

canmatrix/src/canmatrix/canmatrix.py

Line 1090 in eb6d4f4

def decode(self, data):

and Frame.encode

canmatrix/src/canmatrix/canmatrix.py

Line 987 in eb6d4f4

def encode(self, data=None):

gianpyc · 2019-03-21T08:51:15Z

Thanks for your patient too :)
Still some errors to fix like:

BA_ "Description" BU_ ECU "Some Some Some Text";
BA_ "Description" BU_ ECU "Some Some Text";
BA_ "Description" BU_ ECU "Some Text";
BA_ "Description" BU_ ECU_NAME "Some Text";

Also thanks for the encode/decode functions. Is there a way to invoke using command line? Or in case is there a python example in which a I can load my dbc and encode signals of an ECU?

ebroecker · 2019-03-21T19:50:56Z

ok, I think, I made it worse with one of the first commits and now have to repair it again...
Again a new fix for your issue...

for encoding/decoding: What is your usecase, can you describe a bit, maybe I create a feature for you.
otherwise decoding/encoding is not that complicated.
Have a look at
https://github.com/ebroecker/canmatrix/blob/development/src/canmatrix/tests/test_frame_decoding.py
and
https://github.com/ebroecker/canmatrix/blob/development/src/canmatrix/tests/test_frame_encoding.py

gianpyc · 2019-03-21T20:51:03Z

Fantastic now it works without any errors! Thanks a lot 👍
Do you think to integrate the changes in the main branches?

For encoding my wish is to have a command line script in which I could set as parameter the name of the frame, e.g., testFrame1 in your example dbc, and the script prompts me to insert the value for each signal and at the end it provides me the encoded frame.

Cheers

fix for #330

ebroecker · 2019-03-22T19:12:22Z

yes the changes will be merged to mainline, this will take a few days, because I want to do some review and cleanup.

I also going to create some example tool to create one frame

ebroecker · 2019-03-22T20:27:51Z

I created an example tool in branch cliEncodeOneFrame
usage like:
encodeFrame -f testFrameName nameOfDbc.dbc

gianpyc · 2019-03-22T23:02:13Z

Thanks a lot for encodeFrame cli! It is what I needed. Can I ask you to do a similar thing for the decode, in which I pass the frame in hex format, the dbc and the frame is decoded.

Really thanks

ebroecker · 2019-03-24T20:16:38Z

Added decodeFrameto cliEncodeOneFrame

…) (#332) * Definition-Names can now contain spaces * add test for frame defines with special character * repair regexp for attributes extraction out of dbc #330 * cleanup some dbc regexps

gianpyc · 2019-03-25T12:11:20Z

Thanks a lot for all support you've done

* add example for encoding frame with user input (#330) * add decodeFrame (#330)

ebroecker · 2019-03-28T14:11:01Z

fixed

ebroecker self-assigned this Mar 17, 2019

ebroecker added a commit that referenced this issue Mar 17, 2019

possible fix for issue #330

3d19bd2

ebroecker added a commit that referenced this issue Mar 17, 2019

Definition-Names can now contain spaces

23e5f3f

fix for #330

ebroecker added a commit that referenced this issue Mar 18, 2019

new attempt for issue #330

8057942

ebroecker added a commit that referenced this issue Mar 19, 2019

allow even more special characters #330

7c5f15c

ebroecker added a commit that referenced this issue Mar 19, 2019

more special characters for defines and attributes #330

0a0a9ed

ebroecker added a commit that referenced this issue Mar 20, 2019

repair regexp for attributes extraction out of dbc #330

348d277

ebroecker added a commit that referenced this issue Mar 21, 2019

further fix for #330

ae3ef6c

ebroecker added a commit that referenced this issue Mar 22, 2019

possible fix for issue #330

f798364

ebroecker added a commit that referenced this issue Mar 22, 2019

Definition-Names can now contain spaces

99967bd

fix for #330

ebroecker added a commit that referenced this issue Mar 22, 2019

new attempt for issue #330

c0ebb25

ebroecker added a commit that referenced this issue Mar 22, 2019

allow even more special characters #330

389d14f

ebroecker added a commit that referenced this issue Mar 22, 2019

more special characters for defines and attributes #330

e4b91e3

ebroecker added a commit that referenced this issue Mar 22, 2019

repair regexp for attributes extraction out of dbc #330

d6bb053

ebroecker added a commit that referenced this issue Mar 22, 2019

further fix for #330

e90334f

ebroecker added a commit that referenced this issue Mar 22, 2019

add example for encoding frame with user input (#330)

1099b14

ebroecker added a commit that referenced this issue Mar 24, 2019

add decodeFrame (#330)

bf7e591

ebroecker added a commit that referenced this issue Mar 28, 2019

Examples for encoding and decoding single frames (#334)

904b289

* add example for encoding frame with user input (#330) * add decodeFrame (#330)

ebroecker closed this as completed Mar 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Errors while parsing DBC #330

Errors while parsing DBC #330

gianpyc commented Mar 16, 2019

altendky commented Mar 16, 2019

gianpyc commented Mar 16, 2019

ebroecker commented Mar 17, 2019

gianpyc commented Mar 17, 2019

ebroecker commented Mar 17, 2019

gianpyc commented Mar 17, 2019

ebroecker commented Mar 17, 2019 •

edited

Loading

ebroecker commented Mar 17, 2019

gianpyc commented Mar 17, 2019

ebroecker commented Mar 17, 2019

gianpyc commented Mar 18, 2019

ebroecker commented Mar 18, 2019

gianpyc commented Mar 19, 2019

ebroecker commented Mar 19, 2019 •

edited

Loading

gianpyc commented Mar 19, 2019

ebroecker commented Mar 19, 2019

ebroecker commented Mar 19, 2019

gianpyc commented Mar 20, 2019

ebroecker commented Mar 20, 2019

gianpyc commented Mar 21, 2019

ebroecker commented Mar 21, 2019

gianpyc commented Mar 21, 2019

ebroecker commented Mar 22, 2019

ebroecker commented Mar 22, 2019

gianpyc commented Mar 22, 2019

ebroecker commented Mar 24, 2019

gianpyc commented Mar 25, 2019

ebroecker commented Mar 28, 2019

Errors while parsing DBC #330

Errors while parsing DBC #330

Comments

gianpyc commented Mar 16, 2019

altendky commented Mar 16, 2019

gianpyc commented Mar 16, 2019

ebroecker commented Mar 17, 2019

gianpyc commented Mar 17, 2019

ebroecker commented Mar 17, 2019

gianpyc commented Mar 17, 2019

ebroecker commented Mar 17, 2019 • edited Loading

ebroecker commented Mar 17, 2019

gianpyc commented Mar 17, 2019

ebroecker commented Mar 17, 2019

gianpyc commented Mar 18, 2019

ebroecker commented Mar 18, 2019

gianpyc commented Mar 19, 2019

ebroecker commented Mar 19, 2019 • edited Loading

gianpyc commented Mar 19, 2019

ebroecker commented Mar 19, 2019

ebroecker commented Mar 19, 2019

gianpyc commented Mar 20, 2019

ebroecker commented Mar 20, 2019

gianpyc commented Mar 21, 2019

ebroecker commented Mar 21, 2019

gianpyc commented Mar 21, 2019

ebroecker commented Mar 22, 2019

ebroecker commented Mar 22, 2019

gianpyc commented Mar 22, 2019

ebroecker commented Mar 24, 2019

gianpyc commented Mar 25, 2019

ebroecker commented Mar 28, 2019

ebroecker commented Mar 17, 2019 •

edited

Loading

ebroecker commented Mar 19, 2019 •

edited

Loading