www.uspto.gov
`
`UNITED STATES DEPARTMENT OF COMMERCE
`United States Patent and TrademarkOffice
`Address; COMMISSIONER FOR PATENTS
`P.O. Box 1450
`Alexandria, Virginia 22313-1450
`
`16/922,369
`
`07/07/2020
`
`Kiyofumi ABE
`
`2020-1984A
`
`8846
`
`UP
`Lind&
`Wenderoth,
`Wenderoth, Lind & Ponack, L.L.P.
`1025 Connecticut Avenue, NW
`Suite 500
`Washington, DC 20036
`
`NIRJHAR, NASIM NAZRUL
`
`2482
`
`PAPER NUMBER
`
`NOTIFICATION DATE
`
`DELIVERY MODE
`
`01/01/2021
`
`ELECTRONIC
`
`Please find below and/or attached an Office communication concerning this application or proceeding.
`
`The time period for reply, if any, is set in the attached communication.
`
`Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
`following e-mail address(es):
`eoa@ wenderoth.com
`kmiller@wenderoth.com
`
`PTOL-90A (Rev. 04/07)
`
`

`

`
`
`Disposition of Claims*
`1-11 is/are pending in the application.
`)
`Claim(s)
`5a) Of the above claim(s) ___ is/are withdrawn from consideration.
`CC) Claim(s)
`is/are allowed.
`Claim(s) 1-11 is/are rejected.
`S)
`) O Claim(s)___is/are objected to.
`C) Claim(s
`are subjectto restriction and/or election requirement
`)
`S)
`* If any claims have been determined allowable, you maybeeligible to benefit from the Patent Prosecution Highway program at a
`participating intellectual property office for the corresponding application. For more information, please see
`http://www.uspto.gov/patents/init_events/pph/index.jsp or send an inquiry to PPHfeedback@uspto.gov.
`
`) )
`
`Application Papers
`10) The specification is objected to by the Examiner.
`11) The drawing(s) filed on 7/7/20 is/are: a)M) accepted or b)L) objected to by the Examiner.
`Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a).
`Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121 (d).
`
`Priority under 35 U.S.C. § 119
`12) Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d)or (f).
`Certified copies:
`_c)¥) None ofthe:
`b)L) Some**
`a)L) All
`1.4) Certified copies of the priority documents have been received.
`2.2 Certified copies of the priority documents have been received in Application No.
`3.4) Copies of the certified copies of the priority documents have been receivedin this National Stage
`application from the International Bureau (PCT Rule 17.2(a)).
`* See the attached detailed Office action for a list of the certified copies not received.
`
`Attachment(s)
`
`1)
`
`Notice of References Cited (PTO-892)
`
`Information Disclosure Statement(s) (PTO/SB/08a and/or PTO/SB/08b)
`2)
`Paper No(s)/Mail Date 7/7/20.
`U.S. Patent and Trademark Office
`
`3) (J Interview Summary (PTO-413)
`Paper No(s)/Mail Date
`(Qj Other:
`
`4)
`
`PTOL-326 (Rev. 11-13)
`
`Office Action Summary
`
`Part of Paper No./Mail Date 20201130
`
`Application No.
`Applicant(s)
`16/922,369
`ABE etal.
`
`Office Action Summary Art Unit|AIA (FITF) StatusExaminer
`NASIM N NIRJHAR
`2482
`Yes
`
`
`
`-- The MAILING DATEofthis communication appears on the cover sheet with the correspondence address --
`Period for Reply
`
`A SHORTENED STATUTORY PERIOD FOR REPLYIS SET TO EXPIRE 3 MONTHS FROM THE MAILING
`DATE OF THIS COMMUNICATION.
`Extensions of time may be available underthe provisions of 37 CFR 1.136(a). In no event, however, may a reply betimely filed after SIX (6) MONTHSfrom the mailing
`date of this communication.
`If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHSfrom the mailing date of this communication.
`-
`- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133}.
`Any reply received by the Office later than three months after the mailing date of this communication, evenif timely filed, may reduce any earned patent term
`adjustment. See 37 CFR 1.704(b).
`
`Status
`
`1) Responsive to communication(s)filed on 7/7/20.
`LC} A declaration(s)/affidavit(s) under 37 CFR 1.130(b) was/werefiled on
`
`2a)(J This action is FINAL. 2b))This action is non-final.
`3) An election was madeby the applicant in responseto a restriction requirement set forth during the interview
`on
`; the restriction requirement and election have been incorporated into this action.
`4\(Z Since this application is in condition for allowance except for formal matters, prosecution as to the merits is
`closed in accordance with the practice under Exparte Quayle, 1935 C.D. 11, 453 O.G. 213.
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 2
`
`DETAILED ACTION
`
`Notice of Pre-AlA or AIA Status
`
`1.
`
`The present application, filed on or after March 16, 2013, is being examined
`
`under the first inventor to file provisions of the AIA.
`
`2.
`
`3.
`
`This communication is responsive to the correspondencefilled on 7/7/20.
`
`Claims 1-11 are presented for examination.
`
`IDS Considerations
`
`The information disclosure statement (IDS) submitted on 7/7/20 is/are being
`
`considered by the examiner as the submission is in compliance with the provisions of 37
`
`CFR 1.97.
`
`Claim Rejections - 35 USC § 103
`
`The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness
`
`rejections set forth in this Office action:
`
`A patent for a claimed invention may not be obtained, notwithstanding that the claimed
`invention is not identically disclosed as setforth in section 102, if the differences between the
`claimed invention and the prior art are such that the claimed invention as a whole would have
`been obvious before the effective filing date of the claimed invention to a person having
`ordinary skill in the art to which the claimed invention pertains. Patentability shall not be
`negated by the mannerin which the invention was made.
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 3
`
`6.
`
`Claims 1-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over
`
`Esenlik (U.S. Pub. No. 20200236387 A1), in view of Kulakov (U.S. Pub. No.
`
`20170208341 A1).
`
`Regarding to claim 1, 6, 10 and 11:
`
`1. Esenlik teach an encoder, comprising: circuitry; and memory, wherein using the
`
`memory,the circuitry: (Esenlik Fig. 2 [0017] In view of the above mentioned problem,
`
`the present disclosure provides motion vector prediction which enables to take into
`
`account the number of accesses to the external memory and the number of samples
`
`which are necessary to be accessible for motion vector refinement of a motion vector for
`
`a coding block)
`
`calculates a cost that is an evaluation value for a current block to be encoded,
`
`(Esenlik [0069] the cost function may be a difference between the current block to be
`
`coded andits prediction block,i.e. the cost function minimizes the residual block. The
`
`minimization of the residual block is based e.g. on calculating a sum of absolute
`
`differences (SAD) between all pixels (samples) of the current block and the candidate
`
`block in the candidate reference picture) for each of a plurality of search points
`
`includedin a first set, (Esenlik [0110] FIG. 8, the initial motion vector pointed to the
`
`center point 810. The search space is gradually constructed around the initial motion
`
`vector position. In the first step, four positions immediately adjacent on the top, bottom,
`
`left and right[first set] to the position 810 pointed to bythe initial motion vector as well
`
`as the position 810 pointed to by the initial motion vector are tested) the plurality of
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 4
`
`search points being a plurality of pixel positions in a reference picture; (Esenlik
`
`[0084] motion vector refinement is performed in a search space which includes
`
`integer pixel positions and fractional pixel positions of a reference picture)
`
`when the basesearch point is determined not to have the lowest cost among the
`
`base search point and the plurality of neighboring search points, calculates a
`
`costthat is the evaluation value for the current block, for each ofa plurality of
`
`search points which spatially neighbor the base search point and are includedin
`
`a secondsetdifferent from thefirst set; (Esenlik [0110] FIG. 8 based on the direction
`
`which results in a lowest cost function among the tested five points, further positions to
`
`be tested are added to the search space. In this example, the lowest cost function could
`
`be seen in the right point and so the search space was extended bythree further points
`
`in the horizontal right direction in the second step. In the second step the lowestcost
`
`function could be seen in right point (with respect to the lowest cost point ofthe first
`
`step), resulting in a further extension of the search space bythree points in the
`
`horizontal right direction.
`
`In the third step the lowest cost function is observed again in
`
`the right point with respect to the lowest cost point of step 2 and results in the extension
`
`of the search space by three more points in the horizontal right direction. According to
`
`the example in FIG. 8, three more steps are performedin the top, top and right
`
`directions in that order)
`
`selects a search point having a lowest cost from among the first set and the
`
`secondset, as a second bestsearch point; and (Esenlik [0165] FIG. 16 finally the
`
`refinement search is applied to K search points (1640) and the best point among the K
`
`search points is selected according to a matching cost function (1650))
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 5
`
`encodesthe current block, using a motion vector corresponding to the first best
`
`search pointor the second bestsearch point. (Esenlik FIG. 3-5 [0170] the motion
`
`vector determination with memory windowlimitation as described above can be
`
`implemented as a part of encoding and/or decoding of a video signal (motion picture))
`
`Esenlik do not explicitly teach determines whether a base search point has a lowest
`
`cost among the base search point and a plurality of neighboring search points
`
`which spatially neighbor the base search point, the base search point and the
`
`plurality of neighboring search points being includedin the first set as the
`
`plurality of search points; when the base search point is determined to have the
`
`lowest cost among the base search point and the plurality of neighboring search
`
`points, selects the base search point as a first best search point;
`
`However Kulakov teach determines whether a base search point has a lowest cost
`
`among the basesearch point and a plurality of neighboring search points which
`
`spatially neighbor the base search point, (Kulakov Fig. 6A [0058] Onceit is
`
`determined wheretheinitial search pattern arrangement is to be centered, the 20 cost
`
`of the block placed at the center is tested, and current BestCostis initialized with this
`
`Cost (MVO). Kulakov [0048] Referring to FIGS. 6-9 c. Steps 2, 4, 8, and 16 all have the
`
`same eight-point diamond pattern B and are numbered according to their step. Thus,
`
`the pattern of step 2 includes points 2-0 to 2-7, and the pattern of step 16 includes
`
`points 16-0 to 16-7) the base search point and the plurality of neighboring search
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 6
`
`points being includedin the first set as the plurality of search points; (Kulakov Fig.
`
`10A [0057] process 1000 maystart by setting or initializing 1002 some initial variables in
`
`an initial or first stage. This may include setting the BestMVto initial motion vector
`
`MV.sub.0. Various alternatives for generating the initial motion vector includes using a
`
`set of predictors such as neighbor MVs whichrefers to using a previously determined
`
`MV on a block(s) adjacent to the 15 one currently being matched, some combination or
`
`median of a number of the neighbor MVs, or an MV from a collocated block ina
`
`previous frame)
`
`when the base search point is determined to have the lowest cost among the
`
`base search point and the plurality of neighboring search points, selects the base
`
`search point as a first best search point; (Kulakov Fig. 10A [0064] the process 1000
`
`then may include comparing 1010 the Cost to the BestCost. If the Cost is smaller than
`
`the BestCost, then new assignment operations 1012 are performed, such that BestCost,
`
`BestMV and BestStep are updated with current values of Cost, motion vector (MV) and
`
`Step)
`
`Examiner's note: Encoding and decoding is done using similar inverse algorithm,
`
`because please refer Esenlik FIG.
`
`1 and FIG. 2.
`
`It would have been obvious before the effective filing date of the claimed
`
`invention to a person having ordinary skill in the art to modify Esenlik, further
`
`incorporating Kulakov in video/camera technology. One would be motivated to do so, to
`
`incorporate determines whether a base search point has a lowest cost among the base
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 7
`
`search point and a plurality of neighboring search points which spatially neighbor the
`
`base search point. This functionality will improve efficiency.
`
`Regarding to claim 2 and 7:
`
`2. Esenlik teach the encoder according to claim 1, and when the end condition is
`
`determinedto be satisfied, encodes the current block, using a motion vector
`
`corresponding to the second best search point selected most recently. (Esenlik
`
`FIG. 3-5 [0170] the motion vector determination with memory windowlimitation as
`
`described above can be implemented as a part of encoding and/or decoding of a video
`
`signal (motion picture))
`
`Esenlik do not explicitly teach wherein the circuitry: when the second best search
`
`point is selected, further determines whether an end condition to end an update
`
`of the base search point is satisfied; when the end condition is determined not to
`
`be satisfied, updates the base search point to the second best search point,
`
`and selects the first best search point based on the base search point updated or
`
`repeats selecting the second best search point;
`
`However Kulakov teach wherein the circuitry: when the second best search point is
`
`selected, (Kulakov Fig. 10A [0064] the process 1000 then may include comparing 1010
`
`the Cost to the BestCost. If the Cost is smaller than the BestCost, then new assignment
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 8
`
`[second best search point] operations 1012 are performed, such that BestCost, BestMV
`
`and BestStep are updated with current values of Cost, motion vector (MV) and Step)
`
`further determines whether an end condition to end an update of the base search
`
`point is satisfied; (Kulakov Fig. 10A [0064] the process 1000 then mayinclude
`
`comparing 1010 the Cost to the BestCost.)
`
`when the end condition is determined not to be satisfied, updates the base
`
`search point to the second best search point, and selects the first best search
`
`point based on the basesearch point updated or repeats selecting the second
`
`best search point; (Kulakov Fig. 10A [0064] the process 1000 then may include
`
`comparing 1010 the Cost to the BestCost. If the Costis smaller than the BestCost, then
`
`new assignment [second best search point] operations 1012 are performed, such that
`
`BestCost, BestMV and BestStep are updated with current values of Cost, motion vector
`
`(MV) and Step. Loop is repeating in Fig. 10A)
`
`Regarding to claim 3 and 8:
`
`3. Esenlik teach the encoder according to claim 1, wherein the circuitry: uses a
`
`pixel position indicated based on a motion vector of an encodedblock,as the
`
`base search point; and (Esenlik [0084] the encoder in its decoder loop may employ
`
`the same refinement to obtain corresponding motion vectors. Motion vector refinement
`
`is performed in a search space which includes integer pixel positions and
`
`fractional pixel positions of a reference picture)
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 9
`
`when the costis calculated for a search point includedin the first set or the
`
`secondset, (Esenlik [0110] FIG. 8 based on the direction which results in a lowest cost
`
`function among the tested five points, further positions to be tested are added to the
`
`search space. In this example, the lowest cost function could be seen in the right point
`
`and so the search space was extended bythree further points in the horizontal right
`
`direction in the secondstep. In the second step the lowest cost function could be seen
`
`in right point (with respect to the lowest costpoint of the first step), resulting in a further
`
`extension of the search space bythree points inthe horizontal right direction. In the
`
`third step the lowest cost function is observed again in the right point with respectto the
`
`lowest cost point of step 2 and results in the extension of the search space by three
`
`more points in the horizontal right direction. According to the example in FIG. 8, three
`
`more steps are performed in the top, top and right directions in that order)
`
`calculates the cost based on (i) an image of a region indicated by the search point
`
`in the reference picture (Esenlik [0091] the function may be sample clipping operation
`
`in combination with sample-wise weighted summation. The template is then used to
`
`perform template matching in the search spaces determined based on MVO and MV1 in
`
`the respective reference pictures 0 and 1. The cost function for determining the best
`
`template match in the respective search spaces is SAD(Template, Block candA’), where
`
`block candA’ is the candidate coding block which is pointed by the candidate MV in the
`
`search space spanned on a position given by the MVO. FIG. 3 illustrates the
`
`determination of the best matching block A’ and the resulting refined motion vector
`
`MV0’)
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 10
`
`and (ii) a base image, and the base image is an image which is obtained from at
`
`least one encoded block (Esenlik FIG. 3-5 [0004] temporal prediction exploits
`
`temporal correlation between pictures, also referred to as frames, of a video. The
`
`temporal prediction is also called inter-prediction, as itis a prediction using the
`
`dependencies between (inter) different video frames. Accordingly, a block
`
`being encoded, also referred to as a current block, is predicted from one or more
`
`previously encodedpicture(s) referred to as a reference picture(s)) which is used for
`
`deriving a motion vector of the current block instead of the current block. (Esenlik
`
`[0009] the motion vector estimation is a computationally complex task in which a
`
`similarity is calculated between the current block [base image] and the corresponding
`
`prediction blocks pointed to by candidate motion vectors in the reference picture.
`
`Typically,
`
`the search region includes MxM samples of the image and each of the
`
`sample position of the MxM candidate positions is tested. The test includes calculation
`
`of a similarity measure between the NxN reference block C and a blockR,located at
`
`the tested candidate position of the search region)
`
`Regarding to claim 4:
`
`4. Esenlik teach the encoder according to claim 1, wherein the circuitry: uses a
`
`pixel position indicated based on a motion vector of an encodedblock, as the
`
`base search point; and (Esenlik [0084] the encoder in its decoder loop may employ
`
`the same refinement to obtain corresponding motion vectors. Motion vector refinement
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 11
`
`is performed in a search space which includes integer pixel positions and
`
`fractional pixel positions of a reference picture)
`
`when the costis calculated for a search point includedin the first set or the
`
`secondset, (Esenlik [0110] FIG. 8 based on the direction which results in a lowest cost
`
`function among the tested five points, further positions to be tested are added to the
`
`search space. In this example, the lowest cost function could be seen in the right point
`
`and so the search space was extended bythree further points in the horizontal right
`
`direction in the secondstep. In the second step the lowest cost function could be seen
`
`in right point (with respect to the lowest cost point of the first step), resulting in a further
`
`extension of the search space bythree points inthe horizontal right direction. In the
`
`third step the lowest cost function is observed again in the right point with respectto the
`
`lowest cost point of step 2 and results in the extension of the search space by three
`
`more points in the horizontal right direction. According to the example in FIG. 8, three
`
`more steps are performed in the top, top and right directions in that order)
`
`calculates the cost based on (i) an image of a region indicated by the search point
`
`in the reference picture (Esenlik [0091] the function may be sample clipping operation
`
`in combination with sample-wise weighted summation. The template is then used to
`
`perform template matching in the search spaces determined based on MVO and MV1 in
`
`the respective reference pictures 0 and 1. The cost function for determining the best
`
`template match in the respective search spaces is SAD(Template, Block candA’), where
`
`block candA’ is the candidate coding block which is pointed by the candidate MV in the
`
`search space spanned onaposition given by the MVO. FIG. 3 illustrates the
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 12
`
`determination of the best matching block A’ and the resulting refined motion vector
`
`MV0')
`
`and (ii) a base image, and the base image is an image of the current block.
`
`(Esenlik [0009] the motion vector estimation is a computationally complex task in which
`
`a similarity is calculated between the current block [base image] and the corresponding
`
`prediction blocks pointed to by candidate motion vectors in the reference picture.
`
`Typically,
`
`the search region includes MxM samples of the image and eachof the
`
`sample position of the MxM candidate positions is tested. The test includes calculation
`
`of a similarity measure between the NxN reference block C and a blockR,located at
`
`the tested candidate position of the search region)
`
`Regarding to claim 5 and 9:
`
`5. Esenlik teach the encoder according to claim 3, wherein when the costis
`
`calculated for the search point includedin the first set or the second set, (Esenlik
`
`[0069] the cost function may be a difference between the current block to be coded and
`
`its prediction block, i.e. the cost function minimizes the residual block. The minimization
`
`of the residual block is based e.g. on calculating a sum of absolute differences (SAD)
`
`between all pixels (samples) of the current block and the candidate block in the
`
`candidate reference picture. Esenlik [0110] FIG. 8, the initial motion vector pointed to
`
`the center point 810. The search space is gradually constructed around the initial
`
`motion vector position. In the first step, four positions immediately adjacent on the top,
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 13
`
`bottom, left and right [first set] to the position 810 pointed to by the initial motion vector
`
`as well as the position 810 pointed to by the initial motion vector are tested)
`
`the circuitry calculates the cost using at least a distortion of the image of the
`
`region with respect to the base image. (Esenlik [0070] However, cost-function may
`
`also be the number of bits necessary to code suchinter-block and/or distortion resulting
`
`from such coding)
`
`Conclusion
`
`Any inquiry concerning this communication or earlier communications from the
`
`examiner should be directed to NASIM N NIRJHAR whose telephone number is (571)
`
`272-3792. The examiner can normally be reached on Monday - Friday, 8am to 5 pm
`
`ET.
`
`if attempts to reach the examiner by telephone are unsuccessful,
`
`the examiner's
`
`supervisor, Christopher Kelley can be reached on (571) 272-7331. The fax phone
`
`number for the organization where this application or proceeding is assigned is (571)
`
`273-8300.
`
`

`

`Application/Control Number: 16/922,369
`Art Unit: 2482
`
`Page 14
`
`Information regarding the status of an application may be obtained from the
`
`Patent Application Information Retrieval (PAIR) system. Status information for
`
`published applications may be obtained from either Private PAIR or Public PAIR.
`
`Status information for unpublished applications is available through Private PAIR only.
`
`For more information about the PAIR system, see http://pair-direct.uspto.gov. Should
`
`you have questions on accessto the Private PAIR system, contact the Electronic
`
`Business Center (EBC) at 866-217-9197(toll-free).
`
`If you would like assistance from a
`
`USPTO Customer Service Representative or access to the automated information
`
`system, call 800-786-9199 (IN USA OR CANADA)or 571-272-1000.
`
`/NASIM N NIRJHAR/
`Primary Examiner, Art Unit 2482
`
`