CAML query IN Clause limiation

Hi All,

I am trying to avoid joining the main document library list with another lookup custom list by first search the lookup custom list and get the document IDs to search them in the document library using IN clause.

The problem with the IN clause is that it does not support more than 500 items, after than an exception is report "unable to handle search query".

I am looking for the recommended solution for this problem.

B.S. I have read an incomplete post "https://social.msdn.microsoft.com/Forums/sharepoint/en-US/350b2e9e-0c50-4887-85d0-b7e8bce9a3b8/caml-query-in-clause-problem?forum=sharepointdevelopmentprevious" that suggests dividing the IN statements into bulks of 500 items, but this did not work with me either.

In my example:

    
<Where>
  <In>
              <FieldRef Name="ProgId" />
               <Values>
                   <Value Type='Number'>6</Value>
-------------------more than 500 times.
                   <Value Type='Number'>10006</Value>
                </Values>
        </In>
</Where>

Thanks,