Currently the "RollupEntityBaseTableName" in the "RollupPropertiesBase" table is 32 characters. Therefore if you have an entity with a base table name longer than 32 characters you will be unable to create Rollup fields for that entity.
This is my situation now for an OnLine customer. Our only solution is to re-create the entire custom entity which is time-consuming. Had we know this would have been an issue obviously we would have created a shorter table name from the start but there is no mention anywhere of this limitation.