United States Patent
`
`[19]
`
`[11] Patent Number:
`
`5,740,801
`
`Branson
`Apr. 21, 1998
`[45] Date of Patent:
`
`USOO574080lA
`
`[54] MANAGING INFORMATION IN AN
`ENDOSCOPY SYSTEM
`
`[76]
`
`Inventor: Philip J. Br-anson. 311 Courthouse RcL,
`Princeton, W. Va. 24740
`
`FOREIGN PATENT DOCUMENTS
`
`3823251 A1
`6000180/5.
`
`1/1989
`1/1994
`
`Japan ............................ G06F 115/40
`Japan .................................. 128/653.1
`
`OTHER PUBLICATIONS
`
`IEEE Engineering in Medicine and Biology; “Image Man-
`agement and Communications for Radiology”; Mar. 1993,
`pp. 70-80.
`Institute of Electrical and Electronics Engineers; “Mumme-
`clia Perspectives for Next Gneeration PAC Systems”; Jun.
`1992, pp. 156-169.
`
`Primary Examiner—Wi]]iam E. Kamm
`Assistant Examiner—Carl H. Layno
`Attorney, Agent, or Fir/n—Adams Law Firm, BA.
`
`[57]
`
`ABSTRACT
`
`A system for acquiring images during a medical procedure
`and using the acquired images includes a storage device for
`storing, for each one of a plurality of users of the system. or
`for each one of a plurality of medical procedures, or for each
`one of a plurality of input or output devices, information that
`indicates one or more processing operations to be performed
`on images obtained by an input device. A system processor
`responds to an identity the user who is currently using the
`system by performing processing operations on the obtained
`images and applying the images to an output device based on
`the stored information that corresponds to the current user.
`
`55 Claims, 21 Drawing Sheets
`
`[21] Appl. No.: 478,865
`
`[22] Filed:
`
`Jun 7, 1995
`
`Related U.S. Application Data
`
`[63] Continuation-in-part of Ser. No. 40,633, Mar. 31, 1993,
`abandoned.
`
`Int. Cl.“ ..................................................... A618 5/103
`[51]
`[52] U.S. C1.
`........................ 128/653.1; 128/920; 128/923
`[58] Field of Search ......................... 364/413.01, 413.02,
`364/413.13; 128/653.1, 921. 923
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`..................... 364/413.13
`7/1979 Daniels et al.
`4,160,906
`5/1984 Andrews et a1.
`....
`.. ..... 364/900
`4,449,195
`3/1989 Komatsu et a1.
`395/6()0
`4,817,050
`9/1991 Danon . .............
`..... 606/10
`5,049,147
`5/1992 Kanno et al.
`358/403
`5,111,306
`5270530 12/1993 Godlewski etal
`250/203.1
`5,319,543
`6/1994 Wilhelm ...........
`.. 364/413.02
`5,361,203
`11/1994 I-Iiyamaetal.
`..
`.. 364/413.13
`5,494,041
`2/1996 W111: .................
`128/653.1
`5,499,109
`3/1996 Mathur etal.
`....... 353/400
`5,509,034
`4/1996 Tanaka
`364/413.13
`5,535,322
`7/1996 Hecht ...................................... 395/155
`
`
`
`10
`
`/
`
`26
`
`
`
`
`
`
`Personal
`Computer
`Program
`
`Preference
`
`Database
`
`
`
`
`18
`
`Canon Exhibit 112
`Page 1
`
`
`
`Image
`
`Output
`Devices
`
`Canon Exhibit 1126
`Page 1
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 1 of 21
`
`5,740,801
`
`(.0
`N
`
`3.
`
`wmu_>mQ
`
`59:0
`
`98
`
`mmmE_
`
`39:0
`
`mm2>mD
`
`«m1..
`
`wmmnmfio
`
`o.
`
`NF
`
`_m:om._mn_
`
`.2:aEoo
`
`E1
`
`mocmhBmi
`
`wmmnflmo
`
`_..O_u_
`
`mmu_>mQ
`
`39:
`
`mag
`
`mmu_>oQ
`
`wmmE_
`
`E9;
`
`wmmnmfio
`
`mm
`
`3
`
`CanonExmbfl112
`Page
`
`Canon Exhibit 1126
`Page 2
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 2 of 21
`
`5,740,801
`
`vu
`
`
`
`umfiuccou>=ow._D
`
`om
`
`
`mmu_>mu_=u_:o
`.Euomv_>_
`
`E.m5_:553am
`
`mm
`
`>r
`
`.o::oE
`
`3
`
`.o.m:_E:___
`
`_
`
`_
`
`mm
`
`..m.c_.m
`
`E_E
`
`Ewzoomm
`
`_xvm
`
`_.35
`
`_Em.m>mn=m
`
`89>
`
`Ea:
`
`venom
`
`EofiBum
`
`mo>
`
`E2m>wn:m
`
`1cvm
`
`51..
`
`E3“.
`
`EmEumm:ms_
`
`Emfimnzw
`
`ova
`
`_S_P..s_
`
`mu_>mo
`
`Eflgmnam
`
`<_os_on_
`
`.mEmn<
`
`F“"——‘_-__—‘""“
`
`_vm2
`
`E2m>m32_$=:Eoo
`.8:mvm4._3.N—la|=ama.o:mammm:
`
`
`
`wEm._uo_mm._mEa0_mc_.£o..o
`new_ow
`
`.€%mBzom
`
`
`
` _.533.350aovmj_
`
`EU.mWDJW
`
`
`
`
`
`25_mm..mEm_..<_m.amu<M
`
`_vm.cvm
`
`3
`
`
`
`50.28(.0283.05
`
`86>
`
`
`
`mmmnmfiomou.::mm_mm.n_nm._:w_.920
`
`
`ESE mu:m._m_mi.._m_.=O_Sq_.m_5.5.__.w_m3m101IIIIInl|.||_anEEE
`
`awn
`
`
`NH__.|....llllll.|II._m8_>oufi_9.29.2.
`wm2_z..._mm.o>m..v_\o—_5a%<a$0.20....
`
`
`
`Canon Exhibit 112
`Page
`
`Canon Exhibit 1126
`Page 3
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`A
`
`
`
`m.5%.
`
`
`
`
`
`53.:_mEw_xwSac.
`
`_m:m_mzoczm.
`
`mcmEm0.Vm
`
`
`
`1,2
`
`hS
`
`12
`
`108.,047’5
`
`..8opco:mm~_.«~._nwi_,_u:>m_S_m_o.320_m:m_w%_.2mo_mc<cammmcmmao_m:<
`
`
`
`
`
`
`
`.._m__o::oo..mco:6coOVm_w:m>coO
`
`
`
`
`mm_mu_tm>Ucm_mEo~_.oIw$_nmEEm.mo.&SE20
`
`
`
`cmm_m:m_m:o=m~_coEo:>m
`
`Canon Exhibit 112
`Page
`
`Canon Exhibit 1126
`Page 4
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 4 of 21
`
`5,740,801
`
`N:
`
`H31.Ba
`
`H34m3mI
`
`54zmmmw
`
`mooo—
`
`_EEou
`
`mesa:vcm29.:o_9.5
`
`
`_mx_adimmc_..mm_5&5.x.2
`
`moP
`
`
`
`oz_._m>m._>.oc._w_2
`
`._.Dn¢H.OEsmmes”.
`
`>mosms_OH
`
`V.0_u_
`
`
`
`.853omu_>
`
`
`
`m:w>oowu_>
`
`$50.0
`
`<G>
`
`on
`
`mam
`
`
`
`5-0+:.23.
`
`Fm
`
`mm
`
`EOON
`
`mEmE
`
`9___€€
`
`9__€._;mom
`
`oEm.__m_<
`
`zuzzm
`
`.2um_mm5.o%_>
`
`u___%_o
`
`Bmmmooi
`
`:._%.8
`
`_o.::oo
`
`9:50
`
`3.8.89
`
`uEnm._0
`
`.ommmoo..Q
`
`E
`
`EP_n_
`
`m.0_n_
`
`Canon Exhibit 112
`Page
`
`Canon Exhibit 1126
`Page 5
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 5 of 21
`
`5,740,801
`
`
`
`22....2.56
`
`m_o:coo
`
`mo:m._2m:n_
`
`mmmnmfio
`
`m.O_u_
`
`CanonExmbfl112
`Page
`
`—S9:
`
`m39:
`
`Canon Exhibit 1126
`Page 6
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 6 of 21
`
`5,740,801
`
`ucmEEoums:
`
`om;
`
`u:mc::oum>mm
`
`uwfiumam22..%mE__s_m_n235._:o=muo_
`
`
`
`
`
`
`
`Banmucmzflua>2umfiomqm
`
`.w::mEE2:SE5.w
`
`
`
`
`
`5:3mums...aEm.m.m83
`
`
`
`x.mE:o_.mo=:m£:m
`
`
`
`mocmmama.3um:_om%_mE.2E3&5..u:mEEoom>mm
`
`
`
`
`
`omu._>m>_._co—
`
`:__~3.m::mmEm.u_
`
`mm.
`
`3m_>mm
`
`
`
`.o:23.mmmumfimbmmmmE_
`
`
`
`__m>m_n_m_uucmo_aEwmm<
`
`.0.m::mmEmE
`
`
`
`mEE..._m::_>
`
`Esm
`
`
`
`Em:2_a._m
`
`mn:mEEoO
`
`_o::o0
`
`wu_>mQ
`
`0.O_n.
`
`
`
`..mE.o.._:m_m_...8.mmwnm.mv
`
`
`
`co_.u<.2530
`
`CanonExmbfl112
`Page
`
`Canon Exhibit 1126
`Page 7
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 7 of 21
`
`5,740,801
`
`3.
`
`
`
`uw.m_uEoo_a._._ow
`
`of
`
`mmmmE_Bmmtmm
`
`
`
`Em:”8___E%_
`
`ucmcm._2amo
`
`.233.oum__num323%..
`
`
`
`:8...:o>w_mama
`
`m>=umnm>mE
`
`.5cozfloccm
`
`:o:m_:a_:mE
`
`
`
` mmzmum__..umumEm:o.:ocE
`
`N3,
`
`of
`
`um::o:.:m::
`
`Eaqmo
`
`E.2N»5o...
`
`6:856%sac29%.0mmmE_
`
`m_..:com2:o_EO
`
`E;
`
`5..2Etompoflww
`
`mucmxxmaE2.
`
`wmmasmu
`
`3:
`
`.0mwmmE_Ezfimo2Um:QEO._Q
`
`Em:
`
`>:m530.058
`
`
`
`.mw._mumsomam
`
`cozmcmfimu
`
`am:3Umaomam
`
`mm.
`
`
`
`ZOFDZDLIDOh
`
`Um:m_:c_
`
`
`
`_mmmmmE_EEanmo.2Etom
`
`
`
`cmsmzcouEm:
`
`€02.m::_umuo.a
`
`55.0.0mam
`
`cc:o:m:m_mmu
`
`>9m:o:moo_
`
`Uwztmqm
`
`.0.mmmnSmu
`
`mocmtmzmi
`
`>2umfiumam
`
`
`
`ficflmoi_aE2xw
`
`N.
`
`0_.u._
`
`CanonExmbfl112
`Page
`
`Canon Exhibit 1126
`Page 8
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 3 of 21
`
`5,740,801
`
`
`
`m::_moaEoo.m>m_:_:§
`
`93.“ucm:o=um_mmEon.
`
`
`
`3__§uucmummamEta
`
`m.2mESmq
`
`
`
`m:_$8oaEooN
`
`
`
`mczmommmmE_
`
`:o:m_oa._mE_
`
`
`
`nm:_own_m~m__.F.O_2:o_...:m>:oo
`
`_mE_E.2
`
`mm.
`
`
`
`.w..:vmoo.n.:om9:m>3
`
`..._:m_mu.5.:_m_ooam
`
`
`
`mmmmE_mc_nEoo>32
`
`m_m:_mco52>5Eo:
`
`593.$8u2:_..n_
`
`
`
`mwmnflmu85535Eu2m:u_mwu
`
`
`
`
`
`vcm$3..m_n_=__..E.0
`
`
`
`mg.Eo:vtoztm8:8.
`
`dmmnflmu
`
`
`
`EO.=uo.uo_ww396.098.
`
`
`
`ommmmnmfinmucwxzwa
`
`
`
`ucmum:_E_m.mu_§=_a
`
`on2mmEE_.o6nE:z
`
`
`
`
`
`.wmmmE_mucmmam.cum
`
`
`
`=aSouscca
`
`m.0_n_
`
`
`
`.o_m.oao3wmmE_2825283°can:3»
`
`
`
`Eozucé:o_.£o::<
`
`
`
`co=mE.2:_co=mu=_Emu_
`
`mucm._w_mE3uozfimqm
`
`mmSo>m_was2umuum
`
`om$mn.s%
`
`mmmmE_:$30»
`
`maE30»
`
`mmmmE_
`
`._:2.:oz
`
`$..§:_
`
`Canon Exhibit 112
`Page
`
`Canon Exhibit 1126
`Page 9
`
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 9 of 21
`
`5,740,801
`
`9:
`
`E:
`
`
`
`ummfiazzm.um_noS:o__om_mm
`
`u=mEo.:m8_m_.Ems_
`
`
`
`ucmmum.._o_oo.Eon_
`
`C0.Zm.C0_._O
`
`.29308mm:3mmmE_couw_muo_
`
`
`
`2830vcmsac.
`
`mc_E_ou6mum:mE_
`
`mu_>mv
`
`2:
`
`
`
`m.omEo_o5.8.:o:moo_.m~_m
`
`
`
`m>_.m_m:mmumzofimczumzm5:.>m_.m>ocmEm2m:_Eooo
`
`
`
`m.0_n_
`
`
`
`mmmE_mo._:om
`
`mm:
`
`
`
`no~_mo._:8.camB830
`
`venomucm
`
`mums:mm>__mo_.mEo_=m
`
`
`
`vcmmco:moo_m>:m_m:
`
`mmwE_2co=m.oc:m5m_mum
`
`._ououmfimfiumm.
`
`
`
`m:_>_mmm.n_.nm>oE
`
`
`
`
`
`dmxon.m_um._no..xo._.
`
`
`
` .u._mon>mxEum.wEw.o.mmm_o__o
`
`
`
`Bfiflmcuvomwn.:ma
`
`
`
`E52.:o:_cmoo..:
`
`
`
`.6.mm:oEcmmbm
`
`.o.m_mao8:8
`
`oom:mE_
`
`N
`0
`—
`
`2.:
`
`8.:mmmE_
`
`Canon Exhibit 112
`Page 1
`
`Canon Exhibit 1126
`Page 10
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 10 of 21
`
`5,740,801
`
`
`
`
`
`mmamzm:m=:26cmIu:mE=:.o=mam:a_mEcozflcwmmam.w:EomE2uEm_w:m=new_um=_Emuc_
`
`
`
`
`
`
`
`
`
`m.m.om.m:o2Em_m>_:ww.m_onE>mumczmumaB
`
`
`
`
`
`Emon>9..2338mEo:umzmzcm
`
`
`
`
`
`0.9.6:m__aEouomRE.59....oE2m>m9:EUmczmum.0nE>m_EmE::8.£oSm:o-coz
`
`
`
`
`
`
`
`
`
`dam;.m>oE”.m._@_._.,.a.xm_aEoo_oE:m
`
`
`
`
`
`
`
`
`
`a266“mm_aEmxmo=_owam.286_:mm:_
`
`
`
`
`
`:6.22mm5cum.m:m:cw.m_B_o
`
`
`
`
`
`.mm=:mmmumczmuam:8Ema?.0mm:
`
`
`
`
`
`.om._=o.0.w§o=m>9.co=um_mm.:o_.mo__n:_u.m:_~_m:m__uEooom2mEmEm>oE
`
`
`
`
`
`
`
`
`
`m:o=oc:_m:_mmmoo._ammmE_E_o::ooncm
`
`
`
`
`
`:o=mo_._nEmcam..m9mE_.0$8cm.0
`
`_.._w:_._>>UCmI
`
`m._w.om.mco
`
`
`
`m_onE>w:m=_:s
`
`8.5%£3,
`
`”._$5.$m._
`
`wm:_:mmE
`
`mmmE_
`
`:o=m_:a_cms_
`
`mm8:60
`
`5%.
`
`E2m>m
`
`can
`
`cmmzow
`
`coach
`
`Efimb
`
`9.0:
`
`Canon Exhibit 112
`11egaP
`
`Canon Exhibit 1126
`Page 11
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 11 of 21
`
`5,740,801
`
`mi.
`
`E.
`
`
`
`2;.2:2=55mm:_omamwmmE_zammu
`Eo:mE.o_c_:¢oEm._2.0:o_m.o>=ou
`
`
`owmmmnmfiu.m=:mmemi
`mu:m.2En_Emimmo
`
`mn—
`
`
`
`
`
`E.ummmmooammmE_
`
`mufiucflm<O>EEO
`
`
`
`wemacsm<mm>
`
`
`
`“co>mEm_u
`
`<O>.55.
`
`m_mum>26
`
`._w55
`
`9.9N2.
`
`at
`
`m..\.—
`
`
`
`mo_.___5:o_€m>coo
`
`_mEw_xw
`
`cmcozaov
`
`:o=mo=:m£:<
`
`88
`
`mi
`
`
`
`9__m.«.$oammmE_
`
`mE£_.ou_m
`
`fimzaam
`
`.2mco=o::m:_
`
`
`
`>_.._%__,..E:E__n_o
`
`
`
`
`
`mi5:....2:mmmE_:_uuuoocm
`
`I.®_n_
`
`
`
`m:zm:__um-=wm.o__mE2:a
`
`_woc.m._m_2acoummmn
`
`
`
`o_Emc.€ucmQauwooa
`
`
`
`w_m>_m:m_mo=wEm£mE
`
`EmumwmE__o
`
`..mhXEn0naC
`an
`211
`1egaP
`
`Canon Exhibit 1126
`Page 12
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 12 of 21
`
`5,740,801
`
`
`
`
`
`
`
`Em_m>m.0Ema.0E=.:o_.om
`
`
`
`
`
`vmzmmu£3,nm;£mEcozum
`
`
`
`v2m..9oE_newufloflmo
`
`
`
`
`
`u__m>fim::B2:_:o=u<
`
`
`
`umfiflmucozomumntomwicozosmo
`
`
`
`
`
`X:
`
`©m—
`
`mo_>mo
`
`m::m:Q._m:.__
`
`oo_>mU
`
`wo:m.m.m:n_
`
`wmmnfimo
`
`3.532
`
`
`
`zzotmmoo_>oQ
`
`
`
`
`
`co:o<ozfimamo.:mo_u69:00
`
`
`
`Amo_>wn:.m>_._O
`
`m_.0_u_
`
`..mhXEn0naC
`an
`211
`1egaP
`
`cm:
`
`20:6.Um:m::_
`
`5.280
`
`uwom:wE_
`
`98am___ea
`
`.0cozomxoo
`
`EmEm>oE
`
`:o_._cmoom_
`
`mo_o>
`
`Canon Exhibit 1126
`Page 13
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 13 of 21
`
`5,740,801
`
`
`
`m_Q:_:E£3,52.2822.
`
`
`
`
`
`8_c%.mucm_x2.mm.mmE_
`
`mmfiofi_o:o=mo__a:ummmwbmv9mmmmE_
`
`mufiom2mmocflmnz
`
`mm__.mama:_o
`
`
`
`
`
`u2__£__8m%__m>__uoE2255
`
`
`
`
`
`m:_:_mEoumm_:cozficmmfin.
`
`
`
`EEmm..o.ouco_.:_omm.:91
`
`
`
`
`
`
`
`macaw8203..EE_m._m._o
`
`m>m:m>O+mmmE_\m.....mmE_
`
`
`
`m_a_::s_;:o>m_mama
`
`
`
`
`
`mc_~_mmmmE_.mmmmE_
`
`
`
`
`
`_m.m:_nEo.o.mc__momucm
`
`xuo.m_oco:_uvm
`
`.mmmmE_
`
`
`
`
`
`a___%...8ca..m__:m
`
`
`
`
`
`dmcmcuEo=m=._mmmE:m>_mmc_
`
`
`
`5%.“:22%52.
`
`
`
`2.0_n_om:
`
`
`
`mco=o::_uEam.m
`
`_m:o_=Ut<
`
`..mhXEn0naC
`an
`211
`1egaP
`
`Canon Exhibit 1126
`Page 14
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 14 of 21
`
`5,740,801
`
`wo_>mo
`
`
`
`
`
`Ewe:mofimcmzHwuou
`
`
`
`Emvcmnmumo_>mo
`
`
`
`
`
`E2m>mu:_omammu_>muSw_u:mEEoo
`
`mucmEE8
`
`3.G_u_
`
`mm.
`
`m_85mmo:m..m.m.a
`
`am:5.>Eo:oxm._.
`
`
`
`$52.:>__mmm
`
`mocmmzmiEm:
`
`Hmmmnmfio
`
`
`
`coEmEmm.Emaoa:_m_2
`
`
`
`_.zmozmnm_oz_wo_>mo
`
`ucmamum5Bo;
`
`GEMEOO
`
`mcozucéB>:...:o__o_o
`
`223°8.>_m:o__o_o
`
`
`
`,o._Umcozucao_mmm
`
`
`
`m:_=.EmzBtm
`
`m=:oE_wE22
`
`32.226:.
`
`ac:26:.
`
`xon255
`
`3.92%
`
`Canon Exhibit 112
`Page 1
`
`Canon Exhibit 1126
`Page 15
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 15 of 21
`
`5,740,801
`
`
`
`
`
`Bamum9:2=3:8___E%_3_>mo
`
`
`
`a:-.m3on_>:mu.o
`
`c2ou-.m3on_
`
`
`
`
`
`.5.mou_>oa.mEmu<
`
`mo_>mU_o:o_tmmc_
`
`__mm2m_nEou
`
`comcozomccoo
`
`bmmmmowc
`
`.mu_mEamum
`
`
`
`a:_§m_..§:3
`
`£25co=mo_::EEou
`
`:9:Ew:EomEEon
`
`30%.0«£9
`
`
`
`m3o__m.23.?
`
`.mS_§:_?3
`
`._oEum__o.Eoum_mu_>mc9:m_o:cou
`
`
`
`mcEomEGo:
`
`
`
`
`
`m_._wEmum._o«mxuom
`
`
`
`
`
`ma>mEwu_>mumEmm
`
`23:>:mE:_own:
`
`om
`
`om
`
`§.§.<
`
`.o_m8Exoow
`
`
`
`co=.Emoom:.2w._o>_.u
`
`
`
`m53:o_.Swaovcm
`
`
`
`”mwo_>mvtamhmfimnm
`
`ms.9uoumo_:2.ou
`
`5co.:ou_mmm
`
`Efimum
`
`.
`
`E.0_..._
`
`
`
`
`
`Emszomo___uoammo_>uo
`
`wosmo
`
`.
`
`QEDEa
`
`Eozzmz
`
`.2amu<
`
`m:_::2:oo
`
`.t..mu:2mo.
`
`3:8>:<
`
`mo_>mu
`
`Canon Exhibit 112
`Page 1
`
`Canon Exhibit 1126
`Page 16
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 16 of 21
`
`5,740,801
`
`mum
`
`3&0cow
`
`E.mo:22mE
`
`ucmmmm.2w
`
`.02.6%
`
`
`
`mwcoowiumooi
`
`
`
`m_ou2oE>u3m
`
`
`
`EQSOumzmmo
`
`mm.=.6moEn_
`
`>:m_uuaw
`
`3:350.
`
`mEmz
`
`.258:/
`
`N_w
`
`
`
`
`
`
`
`__mE:o__m:.m_EoOmu_a::o_.mo:::m.v_vmzscm.
`
`
`
`m_m3Ew;uméuwamm;5E::Boom._mu_um_>_
`
`
`
`
`
`
`
`
`
`o_=own_m.:_m_umn_w
`
`
`
` ucmmacozumm3.5uo..0._m.25Emmcm.u__m>
`
`.0mucmmnmEw._:m_wu
`
`
`
`wo:w._m_mEvmfiomammpmmco2mco
`
`S.O_n_
`
`.69:2>52
`
`oco2>522
`
`>:mE295
`
`
`
`mE|IlmEw:uo_m_:u_mo
`
`_8_.mEm£ms_
`
`_mu_mo._
`
`:mm_oom
`
`
`
`umgmmuEmu.m£o>:<
`
`/23.omaEmco:m_mm
`
`.EmE:o:>cm>::m>.o.mucws_
`
`Bmcozucéucmv—m
`
`
`
`cozootoo.200
`
`oaoowoucm
`
`mv_o:Emu.0m2:_._z<
`
`
`coficmaxm.mm=:oO
`
`:o=m_=qEmE
`
`
`
`co::_omw.:55
`
`.mm=OE_mO
`
`.952=a:_\
`
`.95239:0New
`
`
`
`cozoflmm.....:_E
`
`.ww_mE_.0mm:<
`
`co_.m_oEmE_
`
`.\ow\
`m:_mmwu2n_
`
`
`
`m:o_EOm_n_mmon_
`
`.m7_o__muoz_
`
`>:_o_._m
`
`mum
`
`mmezom82>m_a___:_2
`
`
`
`m.m.mEm.mn_mu_>mo
`
`:o=mu_::EEoO
`
`
`
`:o_.S:m_Eoow.m>>n..mI
`
`mmo_>m0
`
`
`
`
`
`..:_m_um%co_.mE.oE_.5505miumooa
`
`mom
`
`moN
`
`9:3mocmtflma
`
`c_uflmcmfiwu
`
`.mEmz3Few
`
`:o:m:E_Em:
`
`.mE._o_
`
`toaxmommE_
`
`mEo2,mwmn_
`
`mm:_._o£:<
`
`%_m_.28
`
`.3:
`
`o—m
`
`Canon Exhibit 112
`Page 1
`
`Canon Exhibit 1126
`Page 17
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 17 of 21
`
`5,740,801
`
`
`
`co=m:m_muoucmEoi
`
`<_o2un_2u..:§2E.
`
`mu_>mu5
`
`
`
`ou._:om_mEm.xmEum_m:m_mm6
`
`:o:m..E2:_
`
`
`
`__<.EEm>mm:._§..ao
`
`
`
`co_:8___Emu_.m:____
`
`9:Euw::o:wn
`
`
`
`w__m>m_:m.ucm
`
`EmWCOEUC-:
`
`mmmnmfiuoz
`
`9:.0Emu:mawuc_
`
`
`
`mEm.m>m_mE2xm
`
`mifimomass:
`
`.3825:8.
`
`Ema?
`
`omm\\
`
`
`
`Eflmoimmmnflmu
`
`oucwm:=w_xw
`
`
`
`
`
`E_mQcozuci_=:_2
`
`Emzmm:3.6n_oEm_
`
`
`
`:o:mo:_EoE_c_2.E
`
`
`
`:o_.8___E%_.238:
`
`
`
`:o__mE.oE_Eaumooi
`
`
`
`co=m::o_:_._m£o>:<
`
`cozfloccmmama:
`
`:o:mE.o_E
`
`
`
`aefim:o__ao_.cm£:<
`
`
`
`;:___5co_m..w>coo
`
`
`
`.2o:o_S_om2_.mE.o_$2:
`
`
`
`2:._s_%o.._
`
`Ew.m>m
`
`
`
`.59:ou_:O
`
`Emzmxm
`
`
`
`
`
`m:o%__Em.mo._a_mE2xw
`
`t.G_u_
`
`
`
`3%..<_os_on_E0:mmamE_
`
`
`
`dmmnmfiu850.0w_I2uommo-:¢u3.30;d2E_wu
`
`Canon Exhibit 112
`Page 1
`
`Canon Exhibit 1126
`Page 18
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 13 of 21
`
`5,740,801
`
`
`
`Emcmoammmnflmu
`
`oucmmczflxm
`
`
`
`
`
`>=_::m:o_mv:m_w
`
`
`
`Esmoa.6
`
`
`
`
`
`.m>m_.ammncmmmmhxwomwE_
`
`
`
`
`
`E_macozuca:_:s_2:..238:.EmE350
`
`
`
`
`
`
`
`
`
`
`
`E22?:333%.EmzmxmE2m>m
`
`
`
`mmwnmfifi_mEm_:_
`
`wcozocs.
`
`
`
`>._w:u.o_3.0:w_nm__m>ma_Eou_2mum:mE_Em:
`
`
`
`
`
`
`
`
`
`ucm=8.>m_am_u2mE.2cmfimu2momt2:_Em:
`
`
`
`:o=mE.o_:_ummE_
`
`
`
`Bvanum>m_am_um=:m2:8.newuwctotma>830
`
`
`
`
`
`
`
`222.59:2u£a:m_mouEm::_umE_E
`
`
`
`.8955>mm.S.mE26u:oEomm:mEmgamom
`
`
`
`
`
`
`
`m_mmn:8.coumu:_9a<umu:_o:_mEoum_
`
`
`
`
`
`2.0:
`
`
`
`_mE.o_uueowqw:_um>m_am_u\_uo>m_=2$92..
`
`
`
`mu_>mvvmfiuuamco
`
`..mhXEn0naC
`an
`211
`1egaP
`
`Canon Exhibit 1126
`Page 19
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Apr. 21, 1993
`
`Sheet 19 of 21
`
`5,740,301
`
`260 Information Processing
`
`
`
`(Query)
`
`Data Acquisition
`(Procedure)
`
` Signal Processing
`
`280
`
`FIG. 19
`
`Canon Exhibit 112
`Page 2
`
`Canon Exhibit 1126
`Page 20
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 20 of 21
`
`5,740,801
`
`Application Specific Data
`
`E 300
`
`9r:1®filill]@
`
`[El
`
`5.
`
`‘v
`
`3
`
`40
`33° 320
`330
`360
`370
`14 EE—EE
`
`GENERAL and EMM
`
`35
`
`310
`
`Application Specific Functionality
`
`FIG. 20
`
`Canon Exhibit 112
`Page 21
`
`Canon Exhibit 1126
`Page 21
`
`

`
`U.S. Patent
`
`Apr. 21, 1998
`
`Sheet 21 of 21
`
`5,740,801
`
`/
`Structure of an IT record
`
`fl IllIlIIIIllIllIIllIl
`
`The Actual Data
`
`410'
`
`
`
`Numeric iD of this record's owner
`
`Numeric ID of this record's "WHAT"
`
`I
`
`Numeric ID of this record (the its_ id)
`
`400 r
`
`Structure of a WHAT record
`
`)/
`
` lIIIIIIIIIIIIIIIIIII
`
`‘
`
`
`
`
`
`The text name oi what the "WHAT" record specifies
`
`impiementation specific identifier telling how data is decoded
`
`"what_id" of this record's owner (or 0 if none)
`
`Numeric ID of this record (the what __ id)
`
`figmplg figngmg definition
`
`This sample is shown using RAIMA database language. but can be applied to any
`generic database handler.
`
`record what[
`unique key long what__id:
`key
`long what_ owner:
`char what__ name-[32]
`long what_ look
`
`/'1ie|d types '/
`/'“9'd k9Y 7
`'/‘K93’ 07 "5 OWUBT '/
`/‘field name '/
`/‘it's appearance '/
`
`FIG. 21
`
`Canon Exhibit 112
`Page 2
`
`Canon Exhibit 1126
`Page 22
`
`

`
`1
`MANAGING INFORMATION IN AN
`ENDOSCOPY SYSTEM
`
`CROSS REFERENCE TO RELATED
`APPLICKFION
`
`This is a continuation-in-part of U.S. patent application
`Ser. No. 08/040,633. filed Mar. 31, 1993 and abandoned on
`Sep. 6, 1995,
`the entire disclosure of which is hereby
`incorporated herein by reference.
`
`BACKGROUND OF THE INVENTION
`
`This invention relates to managing information in an
`endoscopy system, and in particular to controlling the
`acquisition, processing, storage, and display of endoscopic
`images.
`During an endoscopic surgical procedure, the surgeon
`typically uses a video camera (or other suitable device, such
`as a video arthroscope) to capture images of the surgical site.
`The images are generally applied to a display device (such
`as a television monitor) for observation. In some cases, the
`images are stored on video tape (by, e.g., a VCR) or are
`converted to digital files for storage in memory or on a disk
`drive. Ditferent physicians often use the video equipment in
`different ways.
`SUMMARY OF THE INVENTION
`
`One general aspect of this invention features a system for
`acquiring images during a medical procedure and using the
`acquired images includes a storage device for storing, for
`each one of a plurality of users of the system, information
`that indicates one or more processing operations to be
`perfonned on images obtained by an input device, and a
`processor that responds to an identity the user who is
`currently using the system by performing processing opera-
`tions on the obtained images and applying the images to an
`output device based on the stored information that corre-
`sponds to the current user.
`The stored information alsoindicates, for each user, a
`configuration of the input device and a configuration of the
`output device. During operation. the input and output device
`configurations are established in response to the identity of
`the current user. The configuration of the input device
`includes the format in which the device produces the image,
`and the configuration of the output device includes the
`format in which the device uses the image. As a result. the
`system can translate images obtained in one format (e.g.,
`NTSC, RGB, Y-C. PAL, etc.) to another format for display
`by the output device, all according to the user-specified
`information in the storage device.
`The system may include a plurality of input devices and
`multiple output devices. The storage device stores the infor-
`mation for each input device and each output device. During
`operation, the processor selectively processes images from
`the input devices based on the identity of the current user
`and the stored information. Likewise, the processor selec-
`tively applies the processed images to the output devices
`based on the identity of the current user and the stored
`information.
`
`the preference
`According to aspect of the invention.
`database stores, for each one of a plurality of medical
`procedures. information that indicates one or more process-
`ing operations to be performed on images obtained by an
`input device, and the processor responds to the identity of
`the medical procedure that is currently to be performed by
`performing processing operations based on stored informa-
`tion that corresponds to the current medical procedure.
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`5,740,801
`
`2
`
`According to another aspect invention, the preference
`database stores, for each one of a plurality of users or each
`one of a plurality of medical procedures. information that
`indicates a script of designated images, and the processor
`responds to the identity of the current user or the identity of
`a medical procedure that is currently to be performed by
`causing a “scrip ” of images to be captured based on the
`preference information in the database.
`The information indicates, for each user or medical
`procedure, a sequence of images that are to be obtained
`during the procedure. In one embodiment, dining operation,
`the current user is prompted to obtain the images in the
`sequence. This allows the user to follow pre-established
`scripts (stored as directed by the user) to perform tours of the
`surgical site according to the user’s preference or according
`to the requirements of the current surgical procedure.
`According to another aspect of the invention, the prefer-
`ence database stores the preference information as a plural-
`ity of linked records. A first one of the records identifies a
`type of the information, and a second one of the records
`contains data associated with that type of information.
`According to another aspect of the invention, the prefer-
`ence database stores, for each one of a plurality of users or
`medical procedures, respective preference information that
`indicates one or more visual representations that are pre-
`ferred to be applied to the output device or devices together
`with the images obtained by the input device or devices. The
`processor combines the one or more visual representations
`with the images obtained by the input device based on the
`preference information in the preference database and
`applies the images to the output device or devices together
`with the one or more visual representations.
`The one or more visual representations may include text
`that is anti-aliased for improved appearance, graphical
`objects, or an additional image or images that may be
`obtained from an additional input device. The visual repre-
`sentation may possess the attribute of translucency. In cer-
`tain embodiments the visual representations are prompts for
`prompting a user to capture images in a desired sequence.
`The processor may maintain an aspect ratio of the visual
`representations with respect to the images as the images are
`sealed in size. and may maintain the relative appearance of
`the one or more visual representations with respect to the
`images as the format of the images is changed. The proces-
`sor may store the one or more visual representations in
`overlays in order not to affect the images.
`In certain embodiments the visual representations are
`additional images the processor caused to be stored in a
`memory upon receipt of a save command during a medical
`procedure, and the processor may cause the output device or
`devices to display the images as a composite of postage
`stamp size images or as a sequence of full size images. The
`processor may be configured to invoke image processing on,
`or to delete, images selected by the user from among the
`images applied to the output device or devices.
`According to another aspect of the invention, the prefer-
`ence database stores, for each one of a plurality of input
`devices or output devices. infonnafion that indicates one or
`more processing operations to be performed on the images
`obtained by the particular input device or to be applied to the
`particular output device. The processor responds to an
`indication of the identity of the particular input or output
`device by selectively performing processing operations on
`the images based on the information in the preference
`database.
`
`According to another aspect of the invention, the prefer-
`ence database pre-stores. for each one of a plurality of users
`
`Canon Exhibit 112
`Page 2
`
`Canon Exhibit 1126
`Page 23
`
`

`
`5,740,801
`
`3
`or medical procedures. respective preference information
`that indicates a preferred page layout of images. The pro-
`cessor receives an indication of an identity of a user or
`medical procedure and causes a plurality of images obtained
`by the input device to be arranged in the page layout based
`on the preference information. The preference information
`may indicate a preferred script of designated images that the
`processor causes to be captured based on the preference
`information, the script of images comprising the images to
`be arranged in the page layout.
`According to another aspect of the invention, the proces-
`sor performs a zooming function for increasing size of the
`images on the output device or device, and the processor
`performs the zooming function concentrically about a cen-
`tral point of the images. By performing the zooming func-
`tion concentrically about a central point, rather than, for
`example, an upper left hand comer as in certain text pro-
`oessing systems. the processor accommodates the human
`tendency to focus on the center of an image.
`According to another aspect of the invention, the prefer-
`ence database pre-stores, for each of a plurality of users,
`respective preference information that indicates a point
`about which the user prefers to have a zooming function
`performed; The processor performs the zooming function
`concentrically about the point indicated by the preference
`information.
`
`According to another aspect of the invention, the proces-
`sor enhances visual quality of the images by histogram
`adjustment of the images, and adjusts the enhancement of
`visual quality based on the preference information corre-
`sponding to a particular user or medical procedure.
`According to another aspect of the invention, a still frame
`buffer is provided that temporarily stores an image obtained
`by the input device, and a memory is provided that stores
`images obtained by the input device on a long-terrn basis.
`When the processor receives a freeze command it freezes in
`the still frame buifer an image obtained by the input device,
`and when the processor receives a save command it saves a
`frozen image in the memory.
`In certain embodiments, when the processor receives a
`live command it causes the output device to display live
`images obtained by the input device. The processor can
`toggle between frozen and live images until a desired image
`is frozen and then saved in the memory.
`According to another aspect of the invention, the prefer-
`ence database pre-stores, for each of a plurality of users or
`medical procedures, respective preference information that
`indicates a preferred conversion between a format in which
`the input device obtains the images and a format in which
`the output device communicates the image data to the
`medical practitioner. The processor translates the images
`between the two formats based on the preference informa-
`tion.
`
`According to another aspect of the invention, the proces-
`sor performs dynamic mathematical analysis of image data
`received from the input device. Based on the dynamic
`mathematical analysis of the image data,
`the processor
`automatically converts a format in which the input device
`obtains the images into a format in which the output device
`communicates the image data to the medical practitioner.
`According to another aspect of the invention, the proces-
`sor reduces interlaced noise on an interlaced video display
`device by reducing motion artifacts due to motion that
`occurs between acquisition of fields of video that comprise
`a video frame. At least one pixel in at least one line of a first
`field of video (preferably two pixels in two respective lines
`
`4
`
`of the first field) is compared with at least one pixel in at
`least one line of a second field of Video that is interlaced with
`the first field on the video display device. The line or lines
`of the first field are in close proximity to (preferably adjacent
`to) the line of the second field. The pixel or pixels in the first
`field are in close proximity to (preferably adjacent to) the
`pixel in the second field. The processor determining whether
`a dilference between the value of the pixel or pixels in the
`first field (preferably the average of two pixels in the first
`field) and a value of the pixel in the second field exceeds a
`fixed value. If the difference between the values of the pixels
`exceeds the fixed value. the processor replaces the value of
`the pixel in the second field with a value closer to the value
`of the pixel or pixels in the first field (it is preferably
`replaced with the average of the above-mentioned two pixels
`in the first field). If the difference between the values of the
`pixels is less than the fixed value, the processor maintains
`the value of the pixel in the second field. These steps are
`performed for additional pixels in the above-mentioned line
`or lines of the first field and the above-mentioned line of the
`second field, and for additional lines of the first and second
`fields.
`
`As described in more detail in the detailed description, the
`invention provides numerous other features and offers sev-
`eral advantages. Among them are:
`1. a standardized platform for endoscopic video systems
`that allows more rapid evaluation and improvement of
`hardware and functionality.
`2. simple acquisition and storage of still and live images
`from the operating room or endoscopy suite.
`3. digital enhancement of acquired images for presenta-
`tion in printed and film formats.
`4. annotation of acquired images with text and graphics.
`5. formatting and providing composites of multiple
`images, images and stock images, or drawings in a
`manner that provides spatial, logical or other context
`that improves the communicating value of the acquired
`images.
`6. a stand-alone database application that allows indi-
`vidual doctors to maintain and retrieve images and text
`data.
`
`10
`
`15
`
`20
`
`25
`
`35
`
`45
`
`50
`
`55
`
`7. an interface that allows existing ofiice or hospital
`databases to manage image and text information relat-
`ing to endoscopic applications.
`3. allowing existing endoscopic database and endoscopy
`suite management systems (such as Siscope and others)
`to integrate imaging into their applications.
`9. improving the reliability and dependability of endo-
`scopic image and text data by facilitating simple and
`reliable systems for integration of the data with physi-
`cian and facility work flows and hardware constraints.
`In addition, the invention implements a modular archi-.
`tecture that allows improvement by upgrade instead of
`replacement. Regulatory requirements can often be satisfied
`by a modification to an existing filing, as opposed to a de
`novo filing.
`Other features and advantages of the invention will
`become apparent from the following detailed description,
`and from the claims.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`65
`
`FIG. 1 is a block diagram of a system for managing
`endoscopy information according to the present invention.
`FIG. 2 shows one possible configuration of the system of
`FIG. 1.
`
`Canon Exhibit 112
`Page 2
`
`Canon Exhibit 1126
`Page 24
`
`

`
`5,740,801
`
`5
`FIGS. 3 and 4 shows details of a subsystem used in FIG.
`
`2.
`
`FIG. 5 shows two subsystems of FIG. 2 connected in
`parallel for so-called “picture-in-picture” display of images.
`FIG. 6 illustrates a procedure for capturing and storing
`images in the system of FIG. 1.
`FIG. 7 shows a “tour” procedure for capturing images in
`the system of FIG. 1 according to a predefined script.
`FIG. 8 is useful in understanding how the system of FIG.
`1 fonnats printed outputs of images.
`FIG. 9 illustrates a procedure for annotating images
`obtained by the system of FIG. 1 with text and graphics.
`FIG. 10 is useful in understanding a character recognition
`feature of the system of FIG. 1.
`FIG. 11 shows a procedure implemented by the system of
`FIG. ,1 to convert the video format of an input image to a
`difierent video format for use by an output device.
`FIG. 12 is useful in understanding the operation of a
`remote control feature of the system of FIG. 1.
`FIG. 13 shows a procedure performed by the system of
`FIG. 1 for producing slides from the acquired images.
`FIG. 14 shows the device handling architecture of the
`system of FIG. 1.
`FIG. 15 illustrates the use of universal connectivity
`devices (that use, e.g., the PCMCIA standard) in the system
`of FIG. 1.
`
`_
`
`FIG. 16 is useful in understanding the operation of a
`[reference database of the system of FIG. 1.
`FIGS. 17 and 18 illustrate the storage of acquired images
`in an image database of the system of FIG. 1.
`FIG. 19 is a flow chart that shows the operation of the
`system of FIG. 1 from the standpoint of the user.
`FIG. 20 shows modules of a software program that
`controls the overall operation of the system of FIG. 1.
`FIG. 21 is useful in understanding the structure and
`organization of the databases used in the system of FIG. 1.
`DETAILED DESCRIPTION
`
`Referring to FIG. 1, endoscopic procedure management
`system 10 employs a personal computer 12 that executes a
`stored program 14 to configure and manage all devices used
`by a physician during endoscopy. In particular, personal
`computer 12 receives images generated by one or more
`image input devices 16, processes the images according to
`the preferences of the physician performing the endoscopic
`procedure, and transmits the processed images to one or
`more image output devices 18 for display or storage.
`Personal computer 12 obtains information concerning the
`physician’s preferences about the processing, display, and
`storage of the images from a preference database 20. As
`described in more detail below, preference database 20 is
`organized by physician—that is, the preferences of each
`physician recognized by system 10 as to the image process-
`ing and the configuration and operation of image input
`devices 16 and image output devices 18 are stored in
`preference database 20 according to the identity of the
`physician and according to the various procedures per-
`formed by the physician. As a result. when a physician logs
`onto system 10 (as described below) and identifies the
`endoscopic procedure that he or she is to perform. host
`computer 12 configures image input devices 16 and image
`output devices 18 and controls the processing of the
`obtained images accordi