Decimal field unexpected ValidationError

Using Django Using the ORM
1

Hey, I defined the following DecimalField in Django model:

test_decimal_field = models.DecimalField(max_digits=9, decimal_places=6)

Using this DecimalField, when Im trying to validate values (using clean() method) I’m getting a validation error when the numbers left of the decimal are not exactly 3 digits (max_digits - decimal_places)

for example:

test_decimal_field.clean(23.123, None)
Out[]: ...ValidationError(['Ensure that there are no more than 6 decimal places.'])

Looking into the clean() method, before validation, it first generates the value - Decimal(‘23.1230000’) - and it looks like ‘0’ digits are added to create a value of the length of max_digits (adding 4 zeros). Since this now has 7 digits after the decimal it then fails validation and I get the validation error.

Is this an expected behavior?

2

This does not surprise me. The conversion from float to decimal is not precise.

If you do this conversion using the native Python Decimal class, it helps show what the situation is:

In [14]: Decimal(23.123)
Out[14]: Decimal('23.123000000000001108446667785756289958953857421875')

In [15]: Decimal('23.123')
Out[15]: Decimal('23.123')

It is this string that is being converted to decimal, and failing the field constraints. So it is not padding the original value, it is converting the specified float to its full representation, which is then being truncated to the 9 digits before attempting to be assigned to the field.

When dealing with precise floating point values, you’ll want to supply the string representation to the field.

For more details on this, see the docs for Decimal objects.

1 Like
3

Thanks! So I actually wondered why this bug closed :confused:
https://code.djangoproject.com/ticket/30290

4

I don’t have any insight as to why that happened, but I can agree with what I think the reasoning may have been behind it.

At some level, Django needs to rely upon people knowing Python. It’s not practical for Django to replicate all the Python nuances that may affect Django’s operational characteristics. I can understand that a line needs to be drawn somewhere, and that this can be categorized as being on the Python side of that line.

5

This just bit me as well. I specifically chose a DecimalField because I expected it to handle this issue, rather than using a FloatField. Kind of surprised it doesn’t handle this.

At the very least, it’d be helpful to link in the docs to e.g. django-money for working with currencies because of this issue.

6

I’m curious, handle what exactly?

Python is very up-front about precision and representation issues with floating point values - see 15. Floating-Point Arithmetic: Issues and Limitations — Python 3.12.5 documentation and decimal — Decimal fixed-point and floating-point arithmetic — Python 3.12.5 documentation
The Django docs reference this in the docs for FloatField

Side note: This also isn’t a Django or fundamentally, even a Python issue. This is a computer representational issue applying across languages and systems. (This is one of those things that tend to be taught very early on in any “data processing” course and has been well documented for far longer than I’ve been programming.)

7

Handle the quirks with representations of decimals. I think one of the main reasons people would probably choose it over a float field is for currency, or other use cases where correct representation is important.

I think it’d be helpful to emphasise in the docs that:

  • it’s unsuitable for use as a currency field since it makes no attempt to work around the underlying issues representing floats on CPUs.
  • If users want to store currencies they should use a library like django-money
  • If they require correct representation for non-currency decimals they’ll need to find a suitable replacement

Like I say, for me the confusion arose because I saw FloatField and assumed it wouldn’t provide any such guarantees, but seeing DecimalField I thought it would. A decimal field that can’t accurately represent decimals is only of limited use IMO so its limitations should be explicitly highlighted.

8

The Decimal fields work perfectly. They are ideal for currency.

The issue is that floating point values are not, and the use of floating point fields for currency is never recommended.

If you use Decimal fields consistently, it all works.

The problem is when you try to mix Decimal fields with floating points.

In other words, the problem with:

Is not caused by the Decimal field, the cause of the problem here is the use of the floating point value to convert to the Decimal field.

It does - perfectly.

That would be correct if it were true. But it’s not true. Decimal fields represent decimals exactly.

9

OK great, thanks. I’ll see if I’m using a float somewhere by mistake.

Even your clarification would be helpful, since at least 2 of us have been caught out by inconsistencies, either our mistakes or the field’s. It’d be useful for the docs to highlight exactly what that field does and to suggest issues around inconsistent values are likely caused by casting to floats somewhere.

10

I guess we may need to “agree to disagree” here.

From my perspective, this is a Python issue and not a Django issue.

It’s not practical for Django to document or highlight every Python nuance that can trip someone up along the way. (This is especially true since any particular version of Django can work with different versions of Python - and those nuances could be different by version.)

The docs do. The very first line of text at DecimalField:

A fixed-precision decimal number, represented in Python by a Decimal instance

This refers you directly to the applicable Python docs, which goes into great details about its behavior.

1 Like